Nov 02 08:05:26.799713 ip-10-0-136-103 systemd[1]: Starting Kubernetes Kubelet... Nov 02 08:05:27.260889 ip-10-0-136-103 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:05:27.260889 ip-10-0-136-103 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 08:05:27.260889 ip-10-0-136-103 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:05:27.260889 ip-10-0-136-103 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:05:27.264357 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264269 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:27.264357 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264357 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:27.264357 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264361 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264365 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264368 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264370 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264373 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264375 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264378 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264381 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264383 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264386 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264389 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264391 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264394 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264396 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264399 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264401 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264404 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264406 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264412 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:27.264464 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264415 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264420 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264425 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264427 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264430 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264433 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264435 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264439 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264441 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264443 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264460 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264463 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264466 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264469 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264472 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264475 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264478 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264481 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264484 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:27.264899 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264486 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264489 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264491 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264494 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264496 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264499 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264501 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264504 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264507 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264509 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264512 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264515 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264518 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264520 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264523 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264526 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264528 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264531 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264534 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264536 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:27.265343 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264539 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264542 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264544 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264547 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264549 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264552 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264554 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264558 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264560 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264563 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264566 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264569 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264572 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264576 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264579 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264583 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264586 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264588 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264591 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:27.265831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264593 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264596 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264599 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264601 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264603 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264606 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264608 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264612 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264615 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264618 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264620 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264623 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264626 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264628 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264632 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264634 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264637 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264639 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264642 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264644 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:27.266281 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264647 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264649 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264652 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264654 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264657 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264659 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.264662 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265486 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265493 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265496 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265500 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265502 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265505 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265508 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265511 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265513 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265516 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265518 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265521 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:27.266777 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265524 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265527 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265530 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265533 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265535 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265538 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265541 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265544 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265546 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265549 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265551 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265554 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265557 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265559 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265562 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265565 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265567 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265569 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265572 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265574 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:27.267238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265578 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265580 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265583 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265585 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265588 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265590 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265593 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265595 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265598 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265600 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265603 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265605 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265608 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265613 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265617 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265620 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265623 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265626 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265629 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265632 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:27.267728 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265634 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265637 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265639 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265642 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265644 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265647 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265649 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265652 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265655 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265657 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265660 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265662 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265665 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265668 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265670 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265673 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265675 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265678 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265681 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265683 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:27.268197 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265687 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265689 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265692 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265694 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265697 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265700 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265703 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265705 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265708 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265710 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265713 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265715 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265718 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265720 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265723 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265725 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265728 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265730 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265732 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265735 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:27.268673 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265737 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265740 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265743 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265745 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265747 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265750 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265753 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265756 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265759 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265761 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265764 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265766 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265769 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.265773 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266256 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266262 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266268 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266272 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266274 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:27.269141 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266277 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266280 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266284 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266286 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266289 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266292 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266295 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266298 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266300 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266303 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266305 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266308 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266310 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266313 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266315 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266318 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266320 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266323 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266325 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:27.269596 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266328 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266331 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266334 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266336 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266338 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266341 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266344 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266346 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266349 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266351 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266355 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266357 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266360 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266362 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266365 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266367 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266369 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266374 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266377 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266379 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:27.270057 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266382 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266384 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266387 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266389 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266392 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266394 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266401 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266404 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266406 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266409 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266411 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266414 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266416 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266419 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266422 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266424 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266427 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266429 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266432 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266434 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:27.270566 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266436 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266439 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266442 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266444 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266461 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266464 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266467 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266470 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266472 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266475 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266477 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266480 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266483 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266485 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266488 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266490 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266493 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266496 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266499 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266502 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:27.271036 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266504 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266507 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266509 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266512 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266514 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266517 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266520 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266523 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266525 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266528 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266530 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266532 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266535 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266537 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266540 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266543 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266545 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266548 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266550 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:27.271555 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266553 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266555 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.266558 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267611 2570 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267622 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267628 2570 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267633 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267637 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267641 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267645 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267649 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267654 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267657 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267661 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267664 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267667 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267670 2570 flags.go:64] FLAG: --cgroup-root="" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267673 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267676 2570 flags.go:64] FLAG: --client-ca-file="" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267679 2570 flags.go:64] FLAG: --cloud-provider="external" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267684 2570 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267689 2570 flags.go:64] FLAG: --cluster-domain="" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267692 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267695 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 08:05:27.271993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267698 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267701 2570 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267705 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267708 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267712 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267715 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267718 2570 flags.go:64] FLAG: --contention-profiling="false" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267721 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267724 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267727 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267730 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267734 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267737 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267741 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267744 2570 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267747 2570 flags.go:64] FLAG: --enable-server="true" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267750 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267754 2570 flags.go:64] FLAG: --event-burst="100" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267757 2570 flags.go:64] FLAG: --event-qps="50" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267761 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267765 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267768 2570 flags.go:64] FLAG: --eviction-hard="" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267772 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267775 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267778 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 08:05:27.272585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267781 2570 flags.go:64] FLAG: --eviction-soft="" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267784 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267787 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267790 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267793 2570 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267796 2570 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267799 2570 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267802 2570 flags.go:64] FLAG: --feature-gates="" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267806 2570 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267812 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267815 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267818 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267821 2570 flags.go:64] FLAG: --healthz-port="10248" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267824 2570 flags.go:64] FLAG: --help="false" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267827 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267830 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267833 2570 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267836 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267839 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267843 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267846 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267849 2570 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267851 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 08:05:27.273249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267854 2570 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267857 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267860 2570 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267864 2570 flags.go:64] FLAG: --kube-reserved="" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267867 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267870 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267873 2570 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267876 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267879 2570 flags.go:64] FLAG: --lock-file="" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267882 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267885 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267888 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267893 2570 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267896 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267900 2570 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267903 2570 flags.go:64] FLAG: --logging-format="text" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267905 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267909 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267913 2570 flags.go:64] FLAG: --manifest-url="" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267916 2570 flags.go:64] FLAG: --manifest-url-header="" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267920 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267923 2570 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267927 2570 flags.go:64] FLAG: --max-pods="110" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267930 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267933 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 08:05:27.273820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267936 2570 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267939 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267942 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267945 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267949 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267956 2570 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267960 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267963 2570 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267966 2570 flags.go:64] FLAG: --pod-cidr="" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267969 2570 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267973 2570 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267976 2570 flags.go:64] FLAG: --pods-per-core="0" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267979 2570 flags.go:64] FLAG: --port="10250" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267982 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267984 2570 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01b4be2ff752e38a8" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267988 2570 flags.go:64] FLAG: --qos-reserved="" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267990 2570 flags.go:64] FLAG: --read-only-port="10255" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267993 2570 flags.go:64] FLAG: --register-node="true" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.267996 2570 flags.go:64] FLAG: --register-with-taints="" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268000 2570 flags.go:64] FLAG: --registry-burst="10" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268003 2570 flags.go:64] FLAG: --registry-qps="5" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268006 2570 flags.go:64] FLAG: --reserved-cpus="" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268009 2570 flags.go:64] FLAG: --reserved-memory="" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268013 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268016 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268020 2570 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 08:05:27.274433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268024 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268026 2570 flags.go:64] FLAG: --runonce="false" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268029 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268032 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268035 2570 flags.go:64] FLAG: --seccomp-default="false" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268038 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268041 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268044 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268047 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268050 2570 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268054 2570 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268057 2570 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268059 2570 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268062 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268065 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268068 2570 flags.go:64] FLAG: --system-cgroups="" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268072 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268077 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268080 2570 flags.go:64] FLAG: --tls-cert-file="" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268082 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268087 2570 flags.go:64] FLAG: --tls-min-version="" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268090 2570 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268092 2570 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268096 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268099 2570 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 08:05:27.275417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268102 2570 flags.go:64] FLAG: --v="2" Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268107 2570 flags.go:64] FLAG: --version="false" Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268112 2570 flags.go:64] FLAG: --vmodule="" Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268116 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268120 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268258 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268263 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268266 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268269 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268272 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268274 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268277 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268279 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268282 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268284 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268287 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268289 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268292 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268295 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268297 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:27.276507 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268299 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268302 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268305 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268309 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268311 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268314 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268316 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268319 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268321 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268324 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268327 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268329 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268331 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268334 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268336 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268339 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268341 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268344 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268348 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:27.277331 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268350 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268355 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268359 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268362 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268365 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268367 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268370 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268372 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268375 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268377 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268380 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268382 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268385 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268387 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268390 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268392 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268396 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268399 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268401 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:27.277820 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268404 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268407 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268409 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268411 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268414 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268416 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268419 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268421 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268423 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268426 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268428 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268431 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268438 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268441 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268445 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268461 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268464 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268468 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268471 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:27.278562 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268474 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268476 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268479 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268481 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268484 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268486 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268489 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268491 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268494 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268496 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268500 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268503 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268506 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268508 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268511 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268513 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268516 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268518 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268521 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268523 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:27.279368 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268526 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268529 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268531 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268534 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268537 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268541 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268544 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268546 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268549 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268552 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268554 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268556 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268559 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268562 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.268588 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268623 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:27.280167 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268627 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268630 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268633 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268635 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268638 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268641 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268645 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268648 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268651 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268653 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268656 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268659 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268661 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268664 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268666 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268669 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268671 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268674 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268677 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268679 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:27.280716 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268683 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268685 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268690 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268693 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268695 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268698 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268701 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268703 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268706 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268708 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268711 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268713 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268715 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268718 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268720 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268723 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268725 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268728 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268732 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268734 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:27.281497 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268737 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268739 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268742 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268745 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268747 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268750 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268752 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268754 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268757 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268760 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268762 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268765 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268767 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268770 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268773 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268776 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268779 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268782 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268784 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:27.281978 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268787 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268792 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268796 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268799 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268802 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268804 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268807 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268809 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268812 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268814 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268817 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268819 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268822 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268824 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268827 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268829 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268832 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268834 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268837 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:27.282431 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268839 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268842 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268844 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268847 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268849 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268851 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268854 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268856 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268860 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268862 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268865 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268868 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268871 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268873 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268877 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268881 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268884 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268889 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268892 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:27.283119 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268895 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268897 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268900 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268902 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268904 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268907 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268909 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.268912 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.277121 2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.277143 2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.277946 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.277958 2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278054 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278063 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278068 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278073 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278078 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278083 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278087 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278092 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:27.283791 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278096 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278100 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278104 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278109 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278113 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278117 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278122 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278126 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278130 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278134 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278139 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278144 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278148 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278152 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278157 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278161 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278166 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278170 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278174 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:27.284256 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278178 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278182 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278188 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278193 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278197 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278201 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278205 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278209 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278214 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278218 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278222 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278226 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278230 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278235 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278239 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278243 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278247 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278251 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278255 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278259 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:27.284972 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278264 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278268 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278274 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278280 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278284 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278288 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278293 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278297 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278302 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278306 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278311 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278315 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278319 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278324 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278328 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278334 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278338 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278343 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278347 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:27.285471 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278351 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278356 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278360 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278365 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278369 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278372 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278376 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278382 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278386 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278392 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278398 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278403 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278407 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278411 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278415 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278420 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278424 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278428 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278432 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:27.285969 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278436 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278441 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278445 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278467 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278471 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278475 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278480 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278484 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278488 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278493 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278498 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278503 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278507 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278511 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278515 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278519 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278523 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278527 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278531 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278535 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:27.286416 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278539 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.278582 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278826 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278835 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278840 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278844 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278848 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278852 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278857 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278861 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278865 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278871 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278875 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278879 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278883 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:27.286967 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278889 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278896 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278901 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278905 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278909 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278915 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278919 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278924 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278930 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278934 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278938 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278942 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278946 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278950 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278954 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278958 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278962 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278966 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278971 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278975 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:27.287319 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278979 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278983 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278987 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278991 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278995 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.278999 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279003 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279007 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279014 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279020 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279025 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279030 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279035 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279039 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279043 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279048 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279053 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279057 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279062 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:27.287815 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279066 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279070 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279075 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279079 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279084 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279088 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279092 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279096 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279101 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279105 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279109 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279113 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279117 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279121 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279125 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279129 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279133 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279137 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279141 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:27.288302 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279146 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279151 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279155 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279159 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279163 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279167 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279172 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279176 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279180 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279184 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279189 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279193 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279197 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279201 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279205 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279209 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279213 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279218 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279222 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279226 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:27.289031 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279230 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279234 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279238 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279242 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279246 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279250 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279254 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279259 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279263 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279267 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279271 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279275 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279279 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279283 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279288 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.279327 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:05:27.289776 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279379 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279385 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279390 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279394 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279399 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279403 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279408 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279412 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279416 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279420 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279424 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279428 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279432 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279437 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279443 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279465 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279470 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279474 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279480 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:05:27.290318 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279486 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279490 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279494 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279499 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279503 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279508 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279512 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279516 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279520 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279524 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279529 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279533 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279537 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279541 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279545 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279549 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279553 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279557 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279562 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279566 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:05:27.290790 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279570 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279573 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279578 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279582 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279586 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279590 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279594 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279603 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279609 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279613 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279618 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279623 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279627 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279631 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279635 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279640 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279644 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279648 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279652 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279657 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:05:27.291280 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279660 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279664 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279669 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279673 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279677 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279681 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279685 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279689 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279695 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279700 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279704 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279708 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279712 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279716 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279720 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279724 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279728 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279734 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279738 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:05:27.291767 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279742 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279747 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279752 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279756 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279760 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279764 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279769 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279773 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279777 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279782 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279786 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279790 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279794 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279798 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279802 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279806 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279810 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279814 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279818 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279822 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:05:27.292249 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279826 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279832 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279836 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279840 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279845 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279849 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279854 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:27.279859 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.279948 2570 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.284520 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.285617 2570 server.go:1014] "Starting client certificate rotation" Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.285721 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.285795 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.289815 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 08:05:27.292775 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.291980 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 08:05:27.306463 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.306423 2570 log.go:25] "Validated CRI v1 runtime API" Nov 02 08:05:27.310270 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.310255 2570 log.go:25] "Validated CRI v1 image API" Nov 02 08:05:27.311143 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.311116 2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 08:05:27.316083 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.316062 2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e1628e1a-9bf3-4d60-b390-605570e7a227:/dev/nvme0n1p3 e7888639-533f-44be-b6ce-c17d75aca700:/dev/nvme0n1p4] Nov 02 08:05:27.316154 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.316082 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 08:05:27.321817 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.321695 2570 manager.go:217] Machine: {Timestamp:2026-11-02 08:05:27.320170514 +0000 UTC m=+0.389887976 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100460 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cb327eb33dfebd9749aee3faeaf2b SystemUUID:ec2cb327-eb33-dfeb-d974-9aee3faeaf2b BootID:9bfb9162-f2a2-4ee8-b8f0-659d50b5a507 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e8:16:16:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e8:16:16:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:41:a5:ce:42:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 08:05:27.322768 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.322756 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 08:05:27.322897 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.322883 2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 08:05:27.325937 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.325915 2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 08:05:27.326086 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.325939 2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-103.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 08:05:27.326127 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.326088 2570 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 08:05:27.326127 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.326098 2570 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 08:05:27.326186 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.326131 2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 08:05:27.326930 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.326919 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 08:05:27.328028 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.328019 2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 08:05:27.332241 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.332228 2570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 08:05:27.332356 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.332347 2570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 08:05:27.333204 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.333194 2570 kubelet.go:493] "Attempting to sync node with API server" Nov 02 08:05:27.333249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.333208 2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 08:05:27.333249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.333223 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 08:05:27.333249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.333234 2570 kubelet.go:405] "Adding apiserver pod source" Nov 02 08:05:27.333249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.333242 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 08:05:27.333590 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.333576 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 08:05:27.336487 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.336473 2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 08:05:27.343227 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.343209 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 08:05:27.344438 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.344423 2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 08:05:27.344536 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.344481 2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 08:05:27.344677 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.344664 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 08:05:27.344738 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.344684 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 08:05:27.344738 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.344692 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 08:05:27.344738 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.344702 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 08:05:27.344738 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.344712 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 08:05:27.344738 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.344721 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 08:05:27.345478 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.345467 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 08:05:27.345526 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.345480 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 08:05:27.345526 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.345491 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 08:05:27.345526 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.345500 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 08:05:27.345526 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.345513 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 08:05:27.345663 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.345529 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 08:05:27.346953 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.346942 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 08:05:27.347004 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.346956 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 08:05:27.351521 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.351506 2570 server.go:1263] "Started kubelet" Nov 02 08:05:27.351656 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.351606 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 08:05:27.351720 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.351678 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 08:05:27.351720 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.351594 2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 08:05:27.352340 ip-10-0-136-103 systemd[1]: Started Kubernetes Kubelet. Nov 02 08:05:27.353201 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.353109 2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 08:05:27.353882 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.353862 2570 server.go:317] "Adding debug handlers to kubelet server" Nov 02 08:05:27.358977 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.358955 2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 08:05:27.360232 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.360215 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 08:05:27.360833 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.360816 2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 08:05:27.360926 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.360882 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-103.us-west-2.compute.internal\" not found" Nov 02 08:05:27.361593 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361573 2570 factory.go:55] Registering systemd factory Nov 02 08:05:27.361593 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361586 2570 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 08:05:27.361593 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361597 2570 factory.go:223] Registration of the systemd container factory successfully Nov 02 08:05:27.361744 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361602 2570 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 08:05:27.361744 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361586 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 08:05:27.361744 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361678 2570 reconstruct.go:97] "Volume reconstruction finished" Nov 02 08:05:27.361744 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361687 2570 reconciler.go:29] "Reconciler: start to sync state" Nov 02 08:05:27.361875 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361795 2570 factory.go:153] Registering CRI-O factory Nov 02 08:05:27.361875 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361808 2570 factory.go:223] Registration of the crio container factory successfully Nov 02 08:05:27.361875 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361863 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 08:05:27.361969 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361884 2570 factory.go:103] Registering Raw factory Nov 02 08:05:27.361969 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.361899 2570 manager.go:1199] Started watching for new ooms in manager Nov 02 08:05:27.363059 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.363044 2570 manager.go:319] Starting recovery of all containers Nov 02 08:05:27.372961 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.372935 2570 manager.go:324] Recovery completed Nov 02 08:05:27.378178 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.378166 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:27.379981 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.379966 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:27.380064 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.379994 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:27.380064 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.380005 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:27.380522 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.380510 2570 cpu_manager.go:226] "Starting" policy="none" Nov 02 08:05:27.380585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.380523 2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 08:05:27.380585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.380543 2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 08:05:27.380585 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.380515 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:27.381274 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.381257 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:27.381329 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.381288 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:27.381329 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.381301 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:27.382614 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.382603 2570 policy_none.go:50] "Start" Nov 02 08:05:27.382656 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.382619 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 08:05:27.382656 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.382629 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 08:05:27.384554 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.384539 2570 policy_none.go:44] "Start" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.423214 2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.428101 2570 manager.go:341] "Starting Device Plugin manager" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.428145 2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.428154 2570 server.go:87] "Starting device plugin registration server" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.428485 2570 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.428496 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.428604 2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.428695 2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.428703 2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.429307 2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 08:05:27.441299 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.429379 2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-103.us-west-2.compute.internal\" not found" Nov 02 08:05:27.443948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.443913 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 08:05:27.445285 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.445267 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 08:05:27.445373 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.445289 2570 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 08:05:27.445373 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.445312 2570 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 08:05:27.445373 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.445347 2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 08:05:27.482773 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.482740 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-103.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 08:05:27.484186 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.482740 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-103.us-west-2.compute.internal.18e42bd0ca893620 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-103.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-103.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 08:05:27.351481888 +0000 UTC m=+0.421199350,LastTimestamp:2026-11-02 08:05:27.351481888 +0000 UTC m=+0.421199350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-103.us-west-2.compute.internal,}" Nov 02 08:05:27.529304 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.529227 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:27.530163 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.530146 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:27.530236 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.530177 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:27.530236 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.530187 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:27.530236 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.530212 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.545488 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.545440 2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal"] Nov 02 08:05:27.545591 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.545519 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:27.546439 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.546424 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:27.546550 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.546471 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:27.546550 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.546484 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:27.549009 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.548996 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:27.549157 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.549143 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.549203 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.549173 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:27.549784 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.549767 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:27.549889 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.549784 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:27.549889 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.549799 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:27.549889 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.549808 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:27.549889 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.549810 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:27.549889 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.549818 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:27.551868 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.551853 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.551960 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.551878 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:05:27.552581 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.552552 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:05:27.552581 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.552575 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:05:27.552692 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.552588 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:05:27.568110 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.568087 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 08:05:27.568202 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.568144 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tnm9b" Nov 02 08:05:27.585887 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.585867 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-103.us-west-2.compute.internal\" not found" node="ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.587863 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.587848 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-103.us-west-2.compute.internal\" not found" node="ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.632766 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.632746 2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:27.662270 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.662246 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.662372 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.662274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.662372 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.662296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdbcb27b80ac4a4b19ed5f7d7dc1c25f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"fdbcb27b80ac4a4b19ed5f7d7dc1c25f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.723193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.723167 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:27.723762 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.723742 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:27.724374 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.724355 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:27.724374 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.724367 2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.724507 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:27.724386 2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-103.us-west-2.compute.internal\": node \"ip-10-0-136-103.us-west-2.compute.internal\" not found" Nov 02 08:05:27.761886 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.761860 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.763022 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.762990 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.763022 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.763024 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.763163 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.763041 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdbcb27b80ac4a4b19ed5f7d7dc1c25f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"fdbcb27b80ac4a4b19ed5f7d7dc1c25f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.763163 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.763104 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdbcb27b80ac4a4b19ed5f7d7dc1c25f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"fdbcb27b80ac4a4b19ed5f7d7dc1c25f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.763163 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.763108 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.763163 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.763114 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.818747 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.818670 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:05:27.819918 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.819905 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.871501 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.871474 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:05:27.890897 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.890871 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:27.893060 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:27.893041 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:05:28.285955 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.285866 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 08:05:28.286561 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.285997 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 08:05:28.286561 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.286042 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 08:05:28.286561 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.286063 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 08:05:28.286561 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.286076 2570 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a4c4bab8d27dd4bf6a9cd51fc9ec0bbf-5d8bc6b0cc418bd2.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-103.us-west-2.compute.internal.18e42bd0cc3c0e20\": read tcp 10.0.136.103:33284->3.143.120.91:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-103.us-west-2.compute.internal.18e42bd0cc3c0e20 default 5168 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-103.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-103.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-103.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 08:05:27 +0000 UTC,LastTimestamp:2026-11-02 08:05:27.546437442 +0000 UTC m=+0.616154903,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-103.us-west-2.compute.internal,}" Nov 02 08:05:28.333997 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.333970 2570 apiserver.go:52] "Watching apiserver" Nov 02 08:05:28.361276 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.361246 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 08:05:28.573804 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.573698 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 08:00:28 +0000 UTC" deadline="2028-07-08 02:47:44.63005385 +0000 UTC" Nov 02 08:05:28.573804 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.573739 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14730h42m16.056317612s" Nov 02 08:05:28.579070 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.579041 2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 08:05:28.579394 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.579371 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ndqtq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-rshcn","kube-system/konnectivity-agent-7jfqx","kube-system/global-pull-secret-syncer-xs697","kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal","openshift-image-registry/node-ca-8v9wz","openshift-multus/multus-rclzx","openshift-multus/network-metrics-daemon-zjzfb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm","openshift-cluster-node-tuning-operator/tuned-ppnkn","openshift-multus/multus-additional-cni-plugins-82pzl","openshift-network-operator/iptables-alerter-5g226","openshift-ovn-kubernetes/ovnkube-node-2r44q"] Nov 02 08:05:28.581848 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.581832 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.583989 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.583973 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.585918 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.585904 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:28.585985 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.585962 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:28.587921 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.587910 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:28.589973 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.589940 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.592046 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.592033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.594254 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.594239 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.596891 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.596714 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:28.596891 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.596793 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:28.598994 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.598977 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.601190 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.601174 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.603747 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.603729 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:28.603827 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.603786 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:28.606004 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.605988 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.632627 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.632605 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 08:05:28.632742 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.632640 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 08:05:28.632839 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.632823 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2bt7\"" Nov 02 08:05:28.632892 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.632825 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 08:05:28.633618 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.633600 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:28.633699 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.633603 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:28.633699 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.633641 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rwld\"" Nov 02 08:05:28.637500 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.637483 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 08:05:28.637581 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.637549 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 08:05:28.637652 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.637639 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zrnf9\"" Nov 02 08:05:28.639443 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.639428 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 08:05:28.639511 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.639429 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 08:05:28.639608 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.639596 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 08:05:28.639645 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.639598 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 08:05:28.639720 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.639693 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-c87zx\"" Nov 02 08:05:28.641198 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.641186 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 08:05:28.662947 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.662924 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 08:05:28.667772 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.667750 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-slash\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.667832 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.667788 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-netns\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.667864 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.667834 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-multus\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.667910 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.667859 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-etc-kubernetes\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.667910 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.667881 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-systemd-units\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668004 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.667931 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-ovn\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668004 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.667972 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-netd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668004 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.667997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.668108 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668028 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668108 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-env-overrides\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668108 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668089 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944dd579-cc50-401c-a32f-04055024d5f8-serviceca\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.668108 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668104 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-sys-fs\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.668236 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668118 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80f091be-29ea-430b-a509-e24f1f3169dc-host-slash\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.668236 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668134 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-bin\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668236 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944dd579-cc50-401c-a32f-04055024d5f8-host\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.668236 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668215 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e01da69-1f87-48ec-9edf-b73ccbf0b244-konnectivity-ca\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:28.668346 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668246 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/80f091be-29ea-430b-a509-e24f1f3169dc-iptables-alerter-script\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.668346 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-systemd\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.668346 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668285 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-tuned\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.668346 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668330 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-bin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.668468 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-daemon-config\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.668468 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.668468 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668381 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnb9\" (UniqueName: \"kubernetes.io/projected/80f091be-29ea-430b-a509-e24f1f3169dc-kube-api-access-whnb9\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.668468 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668409 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-modprobe-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.668468 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668437 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-var-lib-kubelet\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.668612 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-binary-copy\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.668612 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:28.668612 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668511 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-systemd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668612 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668544 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-etc-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668612 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668564 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:28.668612 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-netns\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.668770 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668620 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-kubernetes\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.668770 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668770 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668667 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cni-binary-copy\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.668770 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668690 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-socket-dir-parent\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.668770 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668719 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-registration-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.668770 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbtvc\" (UniqueName: \"kubernetes.io/projected/edbc19ec-63f1-446d-989e-97fdc12575f7-kube-api-access-rbtvc\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.668770 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668756 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668773 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-dbus\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668796 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-config\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668818 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8wz\" (UniqueName: \"kubernetes.io/projected/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-kube-api-access-cw8wz\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668833 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysconfig\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668846 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-run\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668861 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-tmp\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668886 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-cnibin\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668902 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-node-log\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668934 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-system-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.668959 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668954 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-multus-certs\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668969 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-conf\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.668989 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-lib-modules\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669004 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-system-cni-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669025 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cnibin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-socket-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669074 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-var-lib-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669092 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-script-lib\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669149 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28h6\" (UniqueName: \"kubernetes.io/projected/275f0ed9-b587-4bda-996c-56f53df75d68-kube-api-access-b28h6\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669208 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj66d\" (UniqueName: \"kubernetes.io/projected/62a68ba4-c84a-47ea-812e-0730bec1645e-kube-api-access-nj66d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.669249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669233 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-hosts-file\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669253 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-kubelet\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-log-socket\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-device-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669358 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6l7h\" (UniqueName: \"kubernetes.io/projected/715adb78-caaa-4665-a368-66bd25143c59-kube-api-access-p6l7h\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669383 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48h5d\" (UniqueName: \"kubernetes.io/projected/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-kube-api-access-48h5d\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669403 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d77fv\" (UniqueName: \"kubernetes.io/projected/944dd579-cc50-401c-a32f-04055024d5f8-kube-api-access-d77fv\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669438 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-k8s-cni-cncf-io\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669469 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-conf-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669484 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-sys\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-host\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.669523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-kubelet-config\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:28.669933 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669526 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/275f0ed9-b587-4bda-996c-56f53df75d68-ovn-node-metrics-cert\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.669933 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669560 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-kubelet\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.669933 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669608 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-tmp-dir\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.669933 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669647 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e01da69-1f87-48ec-9edf-b73ccbf0b244-agent-certs\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:28.669933 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669670 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.669933 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669686 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.669933 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqw98\" (UniqueName: \"kubernetes.io/projected/ee0ef628-8414-4224-82df-4cede1a6d05b-kube-api-access-jqw98\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.669933 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.669750 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-hostroot\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.736361 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.736339 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 08:05:28.736361 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.736339 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 08:05:28.736687 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.736665 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:05:28.736784 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.736707 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 08:05:28.736784 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.736727 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 08:05:28.736784 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.736750 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 08:05:28.737071 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737050 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-87gnn" Nov 02 08:05:28.737132 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.736722 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qmmzq\"" Nov 02 08:05:28.737132 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737087 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 08:05:28.737132 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.736714 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 08:05:28.737389 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737373 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 08:05:28.737527 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737500 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 08:05:28.737592 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737526 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 08:05:28.737592 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737553 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bgzdd\"" Nov 02 08:05:28.737679 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737638 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 08:05:28.739489 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737743 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zgn8x\"" Nov 02 08:05:28.739489 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737767 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-shlrq\"" Nov 02 08:05:28.739489 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737871 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 08:05:28.739489 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.737945 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 08:05:28.739489 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.738025 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwch9\"" Nov 02 08:05:28.739489 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.738085 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 08:05:28.739489 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.738366 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 08:05:28.770441 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.770441 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770442 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnb9\" (UniqueName: \"kubernetes.io/projected/80f091be-29ea-430b-a509-e24f1f3169dc-kube-api-access-whnb9\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.770592 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770482 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-modprobe-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.770592 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-var-lib-kubelet\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.770592 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-binary-copy\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.770592 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770544 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:28.770592 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770565 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-systemd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.770592 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770587 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770591 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-etc-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770627 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-etc-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770605 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-modprobe-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770645 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-netns\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770676 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-var-lib-kubelet\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770698 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-kubernetes\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.770718 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770745 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-kubernetes\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770749 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-netns\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770676 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-systemd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770783 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.770790 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret podName:f4015660-eec1-47d8-8432-de3b74803f6a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:29.270768449 +0000 UTC m=+2.340485901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret") pod "global-pull-secret-syncer-xs697" (UID: "f4015660-eec1-47d8-8432-de3b74803f6a") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770813 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cni-binary-copy\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-socket-dir-parent\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.771116 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-registration-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbtvc\" (UniqueName: \"kubernetes.io/projected/edbc19ec-63f1-446d-989e-97fdc12575f7-kube-api-access-rbtvc\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-dbus\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770939 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-registration-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-config\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770991 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8wz\" (UniqueName: \"kubernetes.io/projected/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-kube-api-access-cw8wz\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.771015 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771018 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysconfig\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771041 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-run\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.771062 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs podName:715adb78-caaa-4665-a368-66bd25143c59 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:29.271045971 +0000 UTC m=+2.340763441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs") pod "network-metrics-daemon-zjzfb" (UID: "715adb78-caaa-4665-a368-66bd25143c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771077 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-run\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.770948 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-socket-dir-parent\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771092 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-dbus\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771141 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysconfig\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-tmp\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771174 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-binary-copy\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.771948 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-cnibin\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771251 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-cnibin\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771263 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-node-log\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-system-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771295 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-node-log\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-multus-certs\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-conf\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771339 2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771358 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-lib-modules\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771383 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-system-cni-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771399 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cni-binary-copy\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771407 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cnibin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-socket-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771475 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-lib-modules\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771485 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-var-lib-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771478 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-system-cni-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771359 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-system-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.772691 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771497 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-conf\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771516 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-multus-certs\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771521 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-script-lib\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771526 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-config\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28h6\" (UniqueName: \"kubernetes.io/projected/275f0ed9-b587-4bda-996c-56f53df75d68-kube-api-access-b28h6\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771554 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cnibin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771568 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-var-lib-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj66d\" (UniqueName: \"kubernetes.io/projected/62a68ba4-c84a-47ea-812e-0730bec1645e-kube-api-access-nj66d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-hosts-file\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771621 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-socket-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771624 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-kubelet\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771692 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-kubelet\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771732 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-hosts-file\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771743 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-log-socket\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771827 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-log-socket\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-device-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771861 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6l7h\" (UniqueName: \"kubernetes.io/projected/715adb78-caaa-4665-a368-66bd25143c59-kube-api-access-p6l7h\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771871 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-device-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771868 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771885 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48h5d\" (UniqueName: \"kubernetes.io/projected/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-kube-api-access-48h5d\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77fv\" (UniqueName: \"kubernetes.io/projected/944dd579-cc50-401c-a32f-04055024d5f8-kube-api-access-d77fv\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771953 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771961 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-script-lib\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.771975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-k8s-cni-cncf-io\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772017 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-k8s-cni-cncf-io\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772033 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772055 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772064 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-conf-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-sys\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772124 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-conf-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772134 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-host\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.773902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772149 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-kubelet-config\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/275f0ed9-b587-4bda-996c-56f53df75d68-ovn-node-metrics-cert\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772165 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-sys\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772180 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-kubelet\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772204 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-host\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772217 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-kubelet\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772209 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-kubelet-config\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772206 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-tmp-dir\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772259 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e01da69-1f87-48ec-9edf-b73ccbf0b244-agent-certs\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772274 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772307 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqw98\" (UniqueName: \"kubernetes.io/projected/ee0ef628-8414-4224-82df-4cede1a6d05b-kube-api-access-jqw98\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772324 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-hostroot\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772343 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-slash\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-netns\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772392 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-multus\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772415 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-etc-kubernetes\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772439 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-systemd-units\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.774555 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772479 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-ovn\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772501 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-tmp-dir\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772504 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772510 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-netd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772549 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-multus\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-etc-kubernetes\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772563 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-systemd-units\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772602 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-hostroot\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772603 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-ovn\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772634 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772639 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-slash\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772642 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-netns\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-env-overrides\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772605 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772669 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-netd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775193 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772672 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944dd579-cc50-401c-a32f-04055024d5f8-serviceca\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772720 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-sys-fs\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80f091be-29ea-430b-a509-e24f1f3169dc-host-slash\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772768 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-bin\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944dd579-cc50-401c-a32f-04055024d5f8-host\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772801 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-sys-fs\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772816 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e01da69-1f87-48ec-9edf-b73ccbf0b244-konnectivity-ca\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772841 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/80f091be-29ea-430b-a509-e24f1f3169dc-iptables-alerter-script\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772855 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80f091be-29ea-430b-a509-e24f1f3169dc-host-slash\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772869 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-systemd\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772891 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-bin\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772892 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-tuned\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-bin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772942 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944dd579-cc50-401c-a32f-04055024d5f8-host\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.772948 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-daemon-config\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.773004 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-env-overrides\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.773316 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.774204 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944dd579-cc50-401c-a32f-04055024d5f8-serviceca\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.775754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.774213 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-bin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.776225 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.774245 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/80f091be-29ea-430b-a509-e24f1f3169dc-iptables-alerter-script\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.776225 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.774280 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-systemd\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.776225 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.775305 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-daemon-config\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.776225 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.775355 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-tmp\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.776225 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.775407 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-tuned\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.776225 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.775513 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e01da69-1f87-48ec-9edf-b73ccbf0b244-konnectivity-ca\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:28.776225 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.775508 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/275f0ed9-b587-4bda-996c-56f53df75d68-ovn-node-metrics-cert\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.776225 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.775582 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e01da69-1f87-48ec-9edf-b73ccbf0b244-agent-certs\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:28.824305 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.824234 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:28.824305 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.824263 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:28.824305 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.824276 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2kv9r for pod openshift-network-diagnostics/network-check-target-rshcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:28.824548 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:28.824355 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r podName:a55d1dbd-a295-4b42-a0fb-b16022237e8c nodeName:}" failed. No retries permitted until 2026-11-02 08:05:29.324335735 +0000 UTC m=+2.394053196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2kv9r" (UniqueName: "kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r") pod "network-check-target-rshcn" (UID: "a55d1dbd-a295-4b42-a0fb-b16022237e8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:28.826966 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.826938 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnb9\" (UniqueName: \"kubernetes.io/projected/80f091be-29ea-430b-a509-e24f1f3169dc-kube-api-access-whnb9\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:28.827159 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.827132 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbtvc\" (UniqueName: \"kubernetes.io/projected/edbc19ec-63f1-446d-989e-97fdc12575f7-kube-api-access-rbtvc\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.827384 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.827360 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8wz\" (UniqueName: \"kubernetes.io/projected/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-kube-api-access-cw8wz\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.828173 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.828153 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77fv\" (UniqueName: \"kubernetes.io/projected/944dd579-cc50-401c-a32f-04055024d5f8-kube-api-access-d77fv\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.828540 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.828516 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj66d\" (UniqueName: \"kubernetes.io/projected/62a68ba4-c84a-47ea-812e-0730bec1645e-kube-api-access-nj66d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.828874 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.828853 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqw98\" (UniqueName: \"kubernetes.io/projected/ee0ef628-8414-4224-82df-4cede1a6d05b-kube-api-access-jqw98\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:28.829126 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.829110 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48h5d\" (UniqueName: \"kubernetes.io/projected/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-kube-api-access-48h5d\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:28.829163 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.829129 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6l7h\" (UniqueName: \"kubernetes.io/projected/715adb78-caaa-4665-a368-66bd25143c59-kube-api-access-p6l7h\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:28.829510 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.829494 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28h6\" (UniqueName: \"kubernetes.io/projected/275f0ed9-b587-4bda-996c-56f53df75d68-kube-api-access-b28h6\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:28.912558 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.912521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:05:28.933182 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.933156 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:05:28.951389 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.951362 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:28.966345 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.966316 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rclzx" Nov 02 08:05:28.971548 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.971526 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:05:28.993198 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:28.993172 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:05:29.012767 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:29.012740 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:05:30.366550 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.366524 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:05:30.378303 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.378282 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:30.408881 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:30.408820 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275f0ed9_b587_4bda_996c_56f53df75d68.slice/crio-c4b11fbdf8c5c21cfa30832d7fa85708793c3b83b5c5590d9a64d94a96fc0695 WatchSource:0}: Error finding container c4b11fbdf8c5c21cfa30832d7fa85708793c3b83b5c5590d9a64d94a96fc0695: Status 404 returned error can't find the container with id c4b11fbdf8c5c21cfa30832d7fa85708793c3b83b5c5590d9a64d94a96fc0695 Nov 02 08:05:30.409706 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:30.409684 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbcb27b80ac4a4b19ed5f7d7dc1c25f.slice/crio-ef179536fb5e26a2e918dd20de15b387edae7710761ab1622a8b9efc87ff5f36 WatchSource:0}: Error finding container ef179536fb5e26a2e918dd20de15b387edae7710761ab1622a8b9efc87ff5f36: Status 404 returned error can't find the container with id ef179536fb5e26a2e918dd20de15b387edae7710761ab1622a8b9efc87ff5f36 Nov 02 08:05:30.410324 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:30.410297 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0551724789a3970575b47d5ece2444c4.slice/crio-646adad95a53d1ac07326d5ca32a8c5f169d7220beab88980809d0bd77845583 WatchSource:0}: Error finding container 646adad95a53d1ac07326d5ca32a8c5f169d7220beab88980809d0bd77845583: Status 404 returned error can't find the container with id 646adad95a53d1ac07326d5ca32a8c5f169d7220beab88980809d0bd77845583 Nov 02 08:05:30.414113 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.414097 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:05:30.504499 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.504466 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:30.598907 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.598879 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:30.599106 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.598941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:30.599106 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:30.599028 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:30.599106 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:30.599060 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:30.599106 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:30.599087 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret podName:f4015660-eec1-47d8-8432-de3b74803f6a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:31.599073153 +0000 UTC m=+3.346066798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret") pod "global-pull-secret-syncer-xs697" (UID: "f4015660-eec1-47d8-8432-de3b74803f6a") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:30.599106 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:30.599107 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs podName:715adb78-caaa-4665-a368-66bd25143c59 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:31.599093378 +0000 UTC m=+3.346087024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs") pod "network-metrics-daemon-zjzfb" (UID: "715adb78-caaa-4665-a368-66bd25143c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:30.649035 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.649007 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:30.699875 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.699802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:30.700016 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:30.699928 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:30.700016 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:30.699940 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:30.700016 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:30.699949 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2kv9r for pod openshift-network-diagnostics/network-check-target-rshcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:30.700016 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:30.699993 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r podName:a55d1dbd-a295-4b42-a0fb-b16022237e8c nodeName:}" failed. No retries permitted until 2026-11-02 08:05:31.699981156 +0000 UTC m=+3.446974802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kv9r" (UniqueName: "kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r") pod "network-check-target-rshcn" (UID: "a55d1dbd-a295-4b42-a0fb-b16022237e8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:30.772897 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.772838 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"fdbcb27b80ac4a4b19ed5f7d7dc1c25f","Type":"ContainerStarted","Data":"ef179536fb5e26a2e918dd20de15b387edae7710761ab1622a8b9efc87ff5f36"} Nov 02 08:05:30.773863 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.773836 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"0551724789a3970575b47d5ece2444c4","Type":"ContainerStarted","Data":"646adad95a53d1ac07326d5ca32a8c5f169d7220beab88980809d0bd77845583"} Nov 02 08:05:30.774820 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:30.774802 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" event={"ID":"275f0ed9-b587-4bda-996c-56f53df75d68","Type":"ContainerStarted","Data":"c4b11fbdf8c5c21cfa30832d7fa85708793c3b83b5c5590d9a64d94a96fc0695"} Nov 02 08:05:31.042709 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.042627 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:05:31.063823 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.063770 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 08:00:29 +0000 UTC" deadline="2028-03-30 17:58:16.686021376 +0000 UTC" Nov 02 08:05:31.063823 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.063816 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12345h52m45.622208939s" Nov 02 08:05:31.145284 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:31.145258 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f091be_29ea_430b_a509_e24f1f3169dc.slice/crio-c82172f79642eea2af2af477954e4b5f6ffb9e0d002a1fb8b2f467ba27b1a474 WatchSource:0}: Error finding container c82172f79642eea2af2af477954e4b5f6ffb9e0d002a1fb8b2f467ba27b1a474: Status 404 returned error can't find the container with id c82172f79642eea2af2af477954e4b5f6ffb9e0d002a1fb8b2f467ba27b1a474 Nov 02 08:05:31.190238 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:31.190212 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944dd579_cc50_401c_a32f_04055024d5f8.slice/crio-e2fac9b613796401bcdc48ac22446d0a347cb436c9ab5bfc7201ce36f70cafa3 WatchSource:0}: Error finding container e2fac9b613796401bcdc48ac22446d0a347cb436c9ab5bfc7201ce36f70cafa3: Status 404 returned error can't find the container with id e2fac9b613796401bcdc48ac22446d0a347cb436c9ab5bfc7201ce36f70cafa3 Nov 02 08:05:31.199360 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:31.199333 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbc19ec_63f1_446d_989e_97fdc12575f7.slice/crio-82f719c36611a6097abc16cf244a1295a4b693ef3dbd19d6211583308d4c0757 WatchSource:0}: Error finding container 82f719c36611a6097abc16cf244a1295a4b693ef3dbd19d6211583308d4c0757: Status 404 returned error can't find the container with id 82f719c36611a6097abc16cf244a1295a4b693ef3dbd19d6211583308d4c0757 Nov 02 08:05:31.288891 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:31.288865 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71abbe8_6a3a_4d0f_9a55_6fe051cf0017.slice/crio-e17baf8b373ad3c96a8fc20e38a90c1f6c90b32662a5e44b5db2fb52bf4903d0 WatchSource:0}: Error finding container e17baf8b373ad3c96a8fc20e38a90c1f6c90b32662a5e44b5db2fb52bf4903d0: Status 404 returned error can't find the container with id e17baf8b373ad3c96a8fc20e38a90c1f6c90b32662a5e44b5db2fb52bf4903d0 Nov 02 08:05:31.294831 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:31.294808 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54dbca4_a028_4ec4_8aae_6edd83aaff7b.slice/crio-ecb4c7cd5ff6e50d991dd72226a5e65034f0b2b66817bb6efdfac434320655fb WatchSource:0}: Error finding container ecb4c7cd5ff6e50d991dd72226a5e65034f0b2b66817bb6efdfac434320655fb: Status 404 returned error can't find the container with id ecb4c7cd5ff6e50d991dd72226a5e65034f0b2b66817bb6efdfac434320655fb Nov 02 08:05:31.324412 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:05:31.324387 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e01da69_1f87_48ec_9edf_b73ccbf0b244.slice/crio-fdc1f1c8a410df54e2792c0e4c8e4a73434731c4d66969e3b87ad47b45a4f88f WatchSource:0}: Error finding container fdc1f1c8a410df54e2792c0e4c8e4a73434731c4d66969e3b87ad47b45a4f88f: Status 404 returned error can't find the container with id fdc1f1c8a410df54e2792c0e4c8e4a73434731c4d66969e3b87ad47b45a4f88f Nov 02 08:05:31.606532 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.606456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:31.606532 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.606500 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:31.606703 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.606599 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:31.606703 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.606604 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:31.606703 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.606648 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs podName:715adb78-caaa-4665-a368-66bd25143c59 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:33.606635423 +0000 UTC m=+5.353629069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs") pod "network-metrics-daemon-zjzfb" (UID: "715adb78-caaa-4665-a368-66bd25143c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:31.606703 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.606660 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret podName:f4015660-eec1-47d8-8432-de3b74803f6a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:33.606654589 +0000 UTC m=+5.353648235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret") pod "global-pull-secret-syncer-xs697" (UID: "f4015660-eec1-47d8-8432-de3b74803f6a") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:31.707901 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.707587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:31.707901 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.707772 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:31.707901 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.707793 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:31.707901 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.707805 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2kv9r for pod openshift-network-diagnostics/network-check-target-rshcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:31.707901 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.707858 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r podName:a55d1dbd-a295-4b42-a0fb-b16022237e8c nodeName:}" failed. No retries permitted until 2026-11-02 08:05:33.707841528 +0000 UTC m=+5.454835188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kv9r" (UniqueName: "kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r") pod "network-check-target-rshcn" (UID: "a55d1dbd-a295-4b42-a0fb-b16022237e8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:31.769644 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.769157 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:31.769644 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.769309 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:31.769644 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.769399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:31.769644 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.769476 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:31.769644 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.769534 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:31.769644 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:31.769598 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:31.779951 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.779919 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" event={"ID":"62a68ba4-c84a-47ea-812e-0730bec1645e","Type":"ContainerStarted","Data":"c549a403bb660ef8c2c4c055050f6c5862b4ef85d040718ed015e7fcbd999733"} Nov 02 08:05:31.781965 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.781934 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ndqtq" event={"ID":"d54dbca4-a028-4ec4-8aae-6edd83aaff7b","Type":"ContainerStarted","Data":"ecb4c7cd5ff6e50d991dd72226a5e65034f0b2b66817bb6efdfac434320655fb"} Nov 02 08:05:31.784918 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.784890 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" event={"ID":"edbc19ec-63f1-446d-989e-97fdc12575f7","Type":"ContainerStarted","Data":"82f719c36611a6097abc16cf244a1295a4b693ef3dbd19d6211583308d4c0757"} Nov 02 08:05:31.786475 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.786417 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5g226" event={"ID":"80f091be-29ea-430b-a509-e24f1f3169dc","Type":"ContainerStarted","Data":"c82172f79642eea2af2af477954e4b5f6ffb9e0d002a1fb8b2f467ba27b1a474"} Nov 02 08:05:31.788076 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.788050 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82pzl" event={"ID":"ee0ef628-8414-4224-82df-4cede1a6d05b","Type":"ContainerStarted","Data":"eddf1c4a5e4f951cd0372023cb686c67eac91db3df8b1d1b1e76ea65c30f82f7"} Nov 02 08:05:31.790011 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.789987 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7jfqx" event={"ID":"8e01da69-1f87-48ec-9edf-b73ccbf0b244","Type":"ContainerStarted","Data":"fdc1f1c8a410df54e2792c0e4c8e4a73434731c4d66969e3b87ad47b45a4f88f"} Nov 02 08:05:31.799461 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.799425 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rclzx" event={"ID":"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017","Type":"ContainerStarted","Data":"e17baf8b373ad3c96a8fc20e38a90c1f6c90b32662a5e44b5db2fb52bf4903d0"} Nov 02 08:05:31.802260 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:31.802229 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8v9wz" event={"ID":"944dd579-cc50-401c-a32f-04055024d5f8","Type":"ContainerStarted","Data":"e2fac9b613796401bcdc48ac22446d0a347cb436c9ab5bfc7201ce36f70cafa3"} Nov 02 08:05:32.064011 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:32.063955 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 08:00:29 +0000 UTC" deadline="2028-07-19 11:30:48.15979774 +0000 UTC" Nov 02 08:05:32.064011 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:32.064006 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15003h25m16.095794688s" Nov 02 08:05:33.625332 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:33.625293 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:33.625810 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:33.625363 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:33.625810 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.625513 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:33.625810 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.625578 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs podName:715adb78-caaa-4665-a368-66bd25143c59 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:37.625558493 +0000 UTC m=+9.372552142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs") pod "network-metrics-daemon-zjzfb" (UID: "715adb78-caaa-4665-a368-66bd25143c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:33.625810 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.625642 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:33.625810 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.625675 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret podName:f4015660-eec1-47d8-8432-de3b74803f6a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:37.625664352 +0000 UTC m=+9.372658001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret") pod "global-pull-secret-syncer-xs697" (UID: "f4015660-eec1-47d8-8432-de3b74803f6a") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:33.726064 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:33.726029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:33.726269 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.726231 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:33.726269 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.726255 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:33.726269 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.726269 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2kv9r for pod openshift-network-diagnostics/network-check-target-rshcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:33.726433 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.726326 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r podName:a55d1dbd-a295-4b42-a0fb-b16022237e8c nodeName:}" failed. No retries permitted until 2026-11-02 08:05:37.726307354 +0000 UTC m=+9.473301002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kv9r" (UniqueName: "kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r") pod "network-check-target-rshcn" (UID: "a55d1dbd-a295-4b42-a0fb-b16022237e8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:33.769053 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:33.769011 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:33.769233 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.769152 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:33.769295 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:33.769282 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:33.769525 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.769366 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:33.769525 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:33.769442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:33.769678 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:33.769554 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:35.769302 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:35.769032 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:35.769781 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:35.769417 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:35.769781 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:35.769282 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:35.769781 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:35.769530 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:35.769781 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:35.769244 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:35.769781 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:35.769614 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:35.837380 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:35.837333 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"fdbcb27b80ac4a4b19ed5f7d7dc1c25f","Type":"ContainerStarted","Data":"feaa6c4e9d9dfd877f01c4371e42994b0c5daf2d9b788555500bdaf7fe88927c"} Nov 02 08:05:35.840290 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:35.840260 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="9bb62d368d9b42d9fa9f625eb72eab80c4866b292d472566bb9c062fe9e6bf60" exitCode=0 Nov 02 08:05:35.840415 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:35.840341 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82pzl" event={"ID":"ee0ef628-8414-4224-82df-4cede1a6d05b","Type":"ContainerDied","Data":"9bb62d368d9b42d9fa9f625eb72eab80c4866b292d472566bb9c062fe9e6bf60"} Nov 02 08:05:35.844476 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:35.844448 2570 generic.go:356] "Generic (PLEG): container finished" podID="0551724789a3970575b47d5ece2444c4" containerID="fd739ec8c0a5113501b1bc521c0a10042a5a83f72afd897bd2eb4855f20e1f5c" exitCode=0 Nov 02 08:05:35.844579 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:35.844492 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"0551724789a3970575b47d5ece2444c4","Type":"ContainerDied","Data":"fd739ec8c0a5113501b1bc521c0a10042a5a83f72afd897bd2eb4855f20e1f5c"} Nov 02 08:05:35.945777 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:35.945714 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" podStartSLOduration=6.9456799270000005 podStartE2EDuration="6.945679927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:35.945463053 +0000 UTC m=+7.692456723" watchObservedRunningTime="2026-11-02 08:05:35.945679927 +0000 UTC m=+7.692673598" Nov 02 08:05:36.859471 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:36.859259 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"0551724789a3970575b47d5ece2444c4","Type":"ContainerStarted","Data":"81fa01985a6af788a9e596f7e7618a2abefb15d88be7601f969ec93f58d60b47"} Nov 02 08:05:36.864965 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:36.864561 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7jfqx" event={"ID":"8e01da69-1f87-48ec-9edf-b73ccbf0b244","Type":"ContainerStarted","Data":"5a606efb576b9af0f25478e0779eeafcbd1f3761cd61a842612e1581679e8206"} Nov 02 08:05:36.968480 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:36.968422 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" podStartSLOduration=7.968402367 podStartE2EDuration="7.968402367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:36.967163343 +0000 UTC m=+8.714157013" watchObservedRunningTime="2026-11-02 08:05:36.968402367 +0000 UTC m=+8.715396226" Nov 02 08:05:37.659578 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:37.659488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:37.659578 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:37.659561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:37.659809 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.659716 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:37.659809 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.659777 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs podName:715adb78-caaa-4665-a368-66bd25143c59 nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.659759825 +0000 UTC m=+17.406753485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs") pod "network-metrics-daemon-zjzfb" (UID: "715adb78-caaa-4665-a368-66bd25143c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:37.660208 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.660168 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:37.660377 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.660246 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret podName:f4015660-eec1-47d8-8432-de3b74803f6a nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.66023178 +0000 UTC m=+17.407225440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret") pod "global-pull-secret-syncer-xs697" (UID: "f4015660-eec1-47d8-8432-de3b74803f6a") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:37.760754 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:37.760711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:37.760974 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.760920 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:37.760974 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.760942 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:37.760974 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.760956 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2kv9r for pod openshift-network-diagnostics/network-check-target-rshcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:37.761193 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.761013 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r podName:a55d1dbd-a295-4b42-a0fb-b16022237e8c nodeName:}" failed. No retries permitted until 2026-11-02 08:05:45.760995868 +0000 UTC m=+17.507989532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kv9r" (UniqueName: "kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r") pod "network-check-target-rshcn" (UID: "a55d1dbd-a295-4b42-a0fb-b16022237e8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:37.770379 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:37.770152 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:37.770379 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.770294 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:37.770379 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:37.770320 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:37.770379 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:37.770353 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:37.770653 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.770456 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:37.770653 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:37.770542 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:37.869527 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:37.869489 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8v9wz" event={"ID":"944dd579-cc50-401c-a32f-04055024d5f8","Type":"ContainerStarted","Data":"5e47bf1bef01c11072372dc172506854f72b660e3aeb81de6a5ac0283e3147c4"} Nov 02 08:05:37.977842 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:37.977788 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7jfqx" podStartSLOduration=4.88208307 podStartE2EDuration="9.977773841s" totalImagesPullingTime="5.095690771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:37.075935294 +0000 UTC m=+8.822928967" watchObservedRunningTime="2026-11-02 08:05:37.977773841 +0000 UTC m=+9.724767509" Nov 02 08:05:38.875753 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:38.875717 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" event={"ID":"edbc19ec-63f1-446d-989e-97fdc12575f7","Type":"ContainerStarted","Data":"ad1d4080445f453adcadb062f4a40aaf9cb07109578b222866821227eefbf281"} Nov 02 08:05:39.769614 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:39.769580 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:39.769748 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:39.769720 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:39.769891 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:39.769873 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:39.770021 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:39.769996 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:39.770089 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:39.770067 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:39.770194 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:39.770136 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:39.784322 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:39.783781 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:39.784839 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:39.784614 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:39.880543 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:39.880505 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rclzx" event={"ID":"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017","Type":"ContainerStarted","Data":"bf8de1dc2b1c2e14498ecfe7472b423c45c9fdf9bf4f2078269cb915f62a72ec"} Nov 02 08:05:39.883634 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:39.883570 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ndqtq" event={"ID":"d54dbca4-a028-4ec4-8aae-6edd83aaff7b","Type":"ContainerStarted","Data":"184a3cf1a1ffb64d52599ff7028fa42865a84a5beaea5c0f7615c78393006176"} Nov 02 08:05:39.892315 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:39.892265 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8v9wz" podStartSLOduration=5.896870173 podStartE2EDuration="11.892250714s" totalImagesPullingTime="5.995380541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:37.978240768 +0000 UTC m=+9.725234436" watchObservedRunningTime="2026-11-02 08:05:39.892250714 +0000 UTC m=+11.639244384" Nov 02 08:05:40.111747 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:40.111488 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ndqtq" podStartSLOduration=2.845461948 podStartE2EDuration="11.111468904s" totalImagesPullingTime="8.266006956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:40.001328096 +0000 UTC m=+11.748321766" watchObservedRunningTime="2026-11-02 08:05:40.111468904 +0000 UTC m=+11.858462574" Nov 02 08:05:40.112142 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:40.112107 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rclzx" podStartSLOduration=3.990490352 podStartE2EDuration="12.112092051s" totalImagesPullingTime="8.121601699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:40.110957046 +0000 UTC m=+11.857950713" watchObservedRunningTime="2026-11-02 08:05:40.112092051 +0000 UTC m=+11.859085722" Nov 02 08:05:40.214610 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:40.214572 2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 08:05:40.767551 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:40.767397 2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T08:05:40.21459387Z","UUID":"9c9abd3e-d477-43b3-931f-4a117a037584","Handler":null,"Name":"","Endpoint":""} Nov 02 08:05:40.769368 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:40.769343 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 08:05:40.769368 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:40.769371 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 08:05:40.886431 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:40.886400 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="083ef05e2a4139e16d8830ab6d2fbf326e1137ca16cc5ffe9e4332ff85444783" exitCode=0 Nov 02 08:05:40.886979 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:40.886506 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82pzl" event={"ID":"ee0ef628-8414-4224-82df-4cede1a6d05b","Type":"ContainerDied","Data":"083ef05e2a4139e16d8830ab6d2fbf326e1137ca16cc5ffe9e4332ff85444783"} Nov 02 08:05:40.890302 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:40.889786 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" event={"ID":"edbc19ec-63f1-446d-989e-97fdc12575f7","Type":"ContainerStarted","Data":"6f0d85342808ee306ccdd0d58007b5033726d2b7106fb680043b43af8428d0ac"} Nov 02 08:05:41.768417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:41.768375 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:41.768417 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:41.768403 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:41.768698 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:41.768551 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:41.768698 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:41.768607 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:41.768698 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:41.768692 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:41.768856 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:41.768784 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:41.894262 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:41.893887 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5g226" event={"ID":"80f091be-29ea-430b-a509-e24f1f3169dc","Type":"ContainerStarted","Data":"f7b40da67ba6fad51c338571bac56c766e462a953f59feb414962d97e55adce8"} Nov 02 08:05:42.006965 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:42.006900 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5g226" podStartSLOduration=5.180291916 podStartE2EDuration="14.006882409s" totalImagesPullingTime="8.826590493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:42.006191032 +0000 UTC m=+13.753184701" watchObservedRunningTime="2026-11-02 08:05:42.006882409 +0000 UTC m=+13.753876079" Nov 02 08:05:42.898646 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:42.898564 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="5d14d86cdc8092d6d5cd7d4ad1ee4a9d92a5c832ba2d313aaf3afb5aed9d5de3" exitCode=0 Nov 02 08:05:42.899371 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:42.898646 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82pzl" event={"ID":"ee0ef628-8414-4224-82df-4cede1a6d05b","Type":"ContainerDied","Data":"5d14d86cdc8092d6d5cd7d4ad1ee4a9d92a5c832ba2d313aaf3afb5aed9d5de3"} Nov 02 08:05:42.900412 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:42.900383 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" event={"ID":"62a68ba4-c84a-47ea-812e-0730bec1645e","Type":"ContainerStarted","Data":"f95793043f535cb0c59f30cce962c01cd6fd92523b871093fb2e121f79eb8a8c"} Nov 02 08:05:42.902898 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:42.902876 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" event={"ID":"edbc19ec-63f1-446d-989e-97fdc12575f7","Type":"ContainerStarted","Data":"98015e68f909d241cb3c48a98dd3f0b652510c9b6e362fe364e9764ae7faaa32"} Nov 02 08:05:43.214533 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:43.214476 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" podStartSLOduration=4.114544103 podStartE2EDuration="15.214461739s" totalImagesPullingTime="11.099917636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:43.213907308 +0000 UTC m=+14.960900975" watchObservedRunningTime="2026-11-02 08:05:43.214461739 +0000 UTC m=+14.961455406" Nov 02 08:05:43.324884 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:43.324835 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" podStartSLOduration=4.609678941 podStartE2EDuration="15.324819598s" totalImagesPullingTime="10.715140657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:43.324450492 +0000 UTC m=+15.071444161" watchObservedRunningTime="2026-11-02 08:05:43.324819598 +0000 UTC m=+15.071813265" Nov 02 08:05:43.768390 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:43.768357 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:43.768575 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:43.768427 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:43.768575 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:43.768526 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:43.768575 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:43.768543 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:43.768716 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:43.768621 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:43.768777 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:43.768732 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:44.240688 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:44.237865 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:44.444281 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:44.444056 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:44.911323 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:44.911292 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82pzl" event={"ID":"ee0ef628-8414-4224-82df-4cede1a6d05b","Type":"ContainerStarted","Data":"c94db155f2d5fbca4dac26294d0b1cab559619958658740023b95d048d0ce808"} Nov 02 08:05:45.723646 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:45.723608 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:45.724087 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:45.723670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:45.724087 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.723792 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:45.724087 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.723843 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs podName:715adb78-caaa-4665-a368-66bd25143c59 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:01.723830377 +0000 UTC m=+33.470824024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs") pod "network-metrics-daemon-zjzfb" (UID: "715adb78-caaa-4665-a368-66bd25143c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:05:45.724245 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.724221 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:45.724373 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.724272 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret podName:f4015660-eec1-47d8-8432-de3b74803f6a nodeName:}" failed. No retries permitted until 2026-11-02 08:06:01.724258921 +0000 UTC m=+33.471252567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret") pod "global-pull-secret-syncer-xs697" (UID: "f4015660-eec1-47d8-8432-de3b74803f6a") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:05:45.768433 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:45.768401 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:45.768596 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:45.768474 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:45.768596 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:45.768492 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:45.768704 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.768596 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:45.768704 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.768678 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:45.768811 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.768776 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:45.824349 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:45.824313 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:45.824530 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.824503 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:05:45.824530 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.824527 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:05:45.824643 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.824540 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2kv9r for pod openshift-network-diagnostics/network-check-target-rshcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:45.824643 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:45.824608 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r podName:a55d1dbd-a295-4b42-a0fb-b16022237e8c nodeName:}" failed. No retries permitted until 2026-11-02 08:06:01.824588888 +0000 UTC m=+33.571582546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kv9r" (UniqueName: "kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r") pod "network-check-target-rshcn" (UID: "a55d1dbd-a295-4b42-a0fb-b16022237e8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:05:45.915939 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:45.915904 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="c94db155f2d5fbca4dac26294d0b1cab559619958658740023b95d048d0ce808" exitCode=0 Nov 02 08:05:45.916075 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:45.915956 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82pzl" event={"ID":"ee0ef628-8414-4224-82df-4cede1a6d05b","Type":"ContainerDied","Data":"c94db155f2d5fbca4dac26294d0b1cab559619958658740023b95d048d0ce808"} Nov 02 08:05:46.341827 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:46.341798 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:46.921140 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:46.920964 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" event={"ID":"275f0ed9-b587-4bda-996c-56f53df75d68","Type":"ContainerStarted","Data":"f1b4d522fabb9b68c451abf5234ed84be4e9984a4663e5cc1fa0f3bf5a4ec768"} Nov 02 08:05:46.921510 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:46.921195 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" event={"ID":"275f0ed9-b587-4bda-996c-56f53df75d68","Type":"ContainerStarted","Data":"889ac48681017812daea5dc297048cb09d7ca3abbd4ccd337b793eb1619169de"} Nov 02 08:05:46.921510 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:46.921227 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" event={"ID":"275f0ed9-b587-4bda-996c-56f53df75d68","Type":"ContainerStarted","Data":"2e70e1c73ed0b25a6cee51cb31438e4327b538bae44c0c56d62460011cb80231"} Nov 02 08:05:46.921510 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:46.921255 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" event={"ID":"275f0ed9-b587-4bda-996c-56f53df75d68","Type":"ContainerStarted","Data":"95502ef1ec9aacaa396fdb8ae2dfba31991c3acfc7549b5cf3bfae3be8ea0847"} Nov 02 08:05:46.921510 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:46.921283 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" event={"ID":"275f0ed9-b587-4bda-996c-56f53df75d68","Type":"ContainerStarted","Data":"e5fbab39a2380ccadbb256fb59ad7d269fa8c16a2399a045e4e91098a37eef89"} Nov 02 08:05:46.921510 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:46.921310 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" event={"ID":"275f0ed9-b587-4bda-996c-56f53df75d68","Type":"ContainerStarted","Data":"4ab0202c8e8ec63422f63385b2c7bb79629ee95ea72ebe7946b0ad049948f4da"} Nov 02 08:05:47.768451 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:47.768415 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:47.768672 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:47.768423 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:47.768672 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:47.768550 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:47.768672 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:47.768434 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:47.768672 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:47.768634 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:47.768890 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:47.768721 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:48.927869 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:48.927831 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" event={"ID":"275f0ed9-b587-4bda-996c-56f53df75d68","Type":"ContainerStarted","Data":"6be6baddc4707bd5c42d974a06c552b69b268ba04d8cfa2ca2613d2e96d54ded"} Nov 02 08:05:48.930489 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:48.930460 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="008db88bdd69666a8bb8848d9fadc04f1f80e9523f89ea4398c69767d48ee44b" exitCode=0 Nov 02 08:05:48.930608 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:48.930511 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82pzl" event={"ID":"ee0ef628-8414-4224-82df-4cede1a6d05b","Type":"ContainerDied","Data":"008db88bdd69666a8bb8848d9fadc04f1f80e9523f89ea4398c69767d48ee44b"} Nov 02 08:05:49.032886 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.032801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:49.238995 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.238910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:49.404463 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.404429 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:49.562700 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.562616 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:49.718963 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.718930 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:49.769077 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.769049 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:49.769229 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.769137 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:49.769229 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.769155 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:49.769307 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:49.769260 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:49.769402 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:49.769379 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:49.769472 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:49.769458 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:49.935993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.935965 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="2cefb1187ca75da12f8d7420bda38bd57b547027d58f4f708ce234fd58f2f10c" exitCode=0 Nov 02 08:05:49.936431 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:49.936011 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82pzl" event={"ID":"ee0ef628-8414-4224-82df-4cede1a6d05b","Type":"ContainerDied","Data":"2cefb1187ca75da12f8d7420bda38bd57b547027d58f4f708ce234fd58f2f10c"} Nov 02 08:05:50.209699 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:50.209611 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:50.693642 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:50.693614 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:50.937805 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:50.937526 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:50.941974 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:50.941950 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" event={"ID":"275f0ed9-b587-4bda-996c-56f53df75d68","Type":"ContainerStarted","Data":"d799b0a7e2c5867b1a2585c24b6100a8475b6a8d213e8705870959446219a302"} Nov 02 08:05:50.942217 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:50.942202 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:50.945356 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:50.945335 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82pzl" event={"ID":"ee0ef628-8414-4224-82df-4cede1a6d05b","Type":"ContainerStarted","Data":"9c843799126380d3094c852e3ec9895b785681ba60891aeac5a1317f9888adab"} Nov 02 08:05:50.958736 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:50.958718 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:51.164765 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:51.162845 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" podStartSLOduration=6.343677902 podStartE2EDuration="22.162827525s" totalImagesPullingTime="15.819149623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:51.162398065 +0000 UTC m=+22.909391734" watchObservedRunningTime="2026-11-02 08:05:51.162827525 +0000 UTC m=+22.909821194" Nov 02 08:05:51.398241 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:51.398120 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-82pzl" podStartSLOduration=9.090725801 podStartE2EDuration="22.398103656s" totalImagesPullingTime="13.307377855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:05:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:51.39793806 +0000 UTC m=+23.144931738" watchObservedRunningTime="2026-11-02 08:05:51.398103656 +0000 UTC m=+23.145097324" Nov 02 08:05:51.656477 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:51.656401 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:51.768835 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:51.768802 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:51.769005 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:51.768884 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:51.769005 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:51.768905 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:51.769005 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:51.768981 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:51.769132 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:51.769048 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:51.769132 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:51.769110 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:51.950025 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:51.949994 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:51.950511 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:51.950078 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:51.974514 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:51.974483 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:05:52.776831 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:52.776804 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:53.174973 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.174943 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:53.384825 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.384740 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:53.394102 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.394081 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ndqtq_d54dbca4-a028-4ec4-8aae-6edd83aaff7b/dns-node-resolver/0.log" Nov 02 08:05:53.458060 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.458025 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rshcn"] Nov 02 08:05:53.458260 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.458152 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:53.458341 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:53.458292 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:53.458840 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.458809 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xs697"] Nov 02 08:05:53.458951 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.458891 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:53.459013 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:53.458984 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:53.459485 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.459457 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zjzfb"] Nov 02 08:05:53.459576 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.459534 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:53.459625 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:53.459609 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:53.547391 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.547353 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:53.773013 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.772979 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:53.847262 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:53.847238 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8v9wz_944dd579-cc50-401c-a32f-04055024d5f8/node-ca/0.log" Nov 02 08:05:54.716345 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:54.716306 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:54.769272 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:54.769243 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:54.769376 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:54.769263 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:54.769457 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:54.769425 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:54.769521 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:54.769463 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:54.769609 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:54.769581 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:54.769776 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:54.769748 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:54.915706 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:54.915671 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:55.080564 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:55.080478 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:55.232686 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:55.232652 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:55.392741 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:55.392658 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:55.539022 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:55.538996 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:56.407929 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:56.407897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:56.557103 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:56.557065 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:56.720010 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:56.719978 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:56.768455 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:56.768428 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:56.768614 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:56.768462 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:56.768614 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:56.768471 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:56.768614 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:56.768567 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:56.768777 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:56.768642 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:56.768777 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:56.768748 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:56.923132 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:56.923097 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:57.106124 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:57.106032 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:57.264454 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:57.264415 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:57.373127 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:57.373033 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:57.373314 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:57.373200 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:05:57.373631 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:57.373603 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:05:57.414863 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:57.414836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:57.563531 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:57.563495 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:58.302333 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:58.302285 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:58.723374 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:58.723348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:58.769591 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:58.769566 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:05:58.769724 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:58.769650 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:05:58.769724 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:58.769696 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:05:58.769808 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:58.769774 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:05:58.769808 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:58.769801 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:05:58.769894 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:05:58.769878 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:05:58.935713 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:58.935527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:59.136535 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:59.136452 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:59.290856 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:59.290822 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:59.459984 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:59.459946 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:59.614483 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:59.614446 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:59.764052 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:59.763971 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:59.938463 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:05:59.938426 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:00.085906 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:00.085820 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:00.289437 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:00.289396 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:00.444922 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:00.444893 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:00.718773 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:00.718686 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:00.768732 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:00.768705 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:00.769221 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:00.768707 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:00.769221 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:00.768833 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:06:00.769221 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:00.768713 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:00.769221 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:00.768879 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:06:00.769221 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:00.768960 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:06:00.916397 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:00.916360 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:01.114488 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:01.114409 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:01.323961 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:01.323921 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:01.517128 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:01.517087 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:01.743845 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:01.743812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:01.744004 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:01.743854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:01.744004 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:01.743939 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:06:01.744004 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:01.743945 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 08:06:01.744004 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:01.743988 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs podName:715adb78-caaa-4665-a368-66bd25143c59 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:33.743973782 +0000 UTC m=+65.490967428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs") pod "network-metrics-daemon-zjzfb" (UID: "715adb78-caaa-4665-a368-66bd25143c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 08:06:01.744004 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:01.744000 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret podName:f4015660-eec1-47d8-8432-de3b74803f6a nodeName:}" failed. No retries permitted until 2026-11-02 08:06:33.743994197 +0000 UTC m=+65.490987843 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret") pod "global-pull-secret-syncer-xs697" (UID: "f4015660-eec1-47d8-8432-de3b74803f6a") : object "kube-system"/"original-pull-secret" not registered Nov 02 08:06:01.845131 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:01.845061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:01.845492 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:01.845212 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 08:06:01.845492 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:01.845226 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 08:06:01.845492 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:01.845234 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2kv9r for pod openshift-network-diagnostics/network-check-target-rshcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:06:01.845492 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:01.845306 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r podName:a55d1dbd-a295-4b42-a0fb-b16022237e8c nodeName:}" failed. No retries permitted until 2026-11-02 08:06:33.845290004 +0000 UTC m=+65.592283668 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2kv9r" (UniqueName: "kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r") pod "network-check-target-rshcn" (UID: "a55d1dbd-a295-4b42-a0fb-b16022237e8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 08:06:02.113517 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:02.113433 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:02.728238 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:02.728201 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:02.769036 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:02.769010 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:02.769135 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:02.769051 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:02.769135 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:02.769018 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:02.769218 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:02.769133 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:06:02.769251 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:02.769218 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:06:02.769288 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:02.769272 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:06:02.943076 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:02.943030 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:03.119395 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:03.119366 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:03.318002 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:03.317916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:03.518318 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:03.518281 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:04.121558 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:04.121528 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:04.339839 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:04.339806 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:04.768895 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:04.768855 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:04.768895 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:04.768876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:04.769101 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:04.768970 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:06:04.769143 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:04.769121 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:06:04.769219 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:04.769206 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:04.769291 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:04.769276 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:06:05.670759 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:05.670724 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:05.873651 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:05.873613 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:06.769389 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:06.769348 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:06.769389 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:06.769374 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:06.769929 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:06.769416 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:06.769929 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:06.769522 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:06:06.769929 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:06.769560 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:06:06.769929 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:06.769631 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:06:08.769745 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:08.769710 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:08.770188 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:08.769797 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:06:08.770188 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:08.769887 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:08.770188 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:08.769978 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:08.770188 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:08.770015 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:06:08.770188 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:08.770054 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:06:10.768430 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:10.768408 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:10.768430 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:10.768422 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:10.768769 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:10.768443 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:10.768769 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:10.768507 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:06:10.768769 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:10.768587 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:06:10.768769 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:10.768669 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:06:12.234039 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:12.234003 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:12.463376 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:12.463346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:12.617867 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:12.617781 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:12.768478 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:12.768436 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:12.768478 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:12.768479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:12.768715 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:12.768583 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjzfb" podUID="715adb78-caaa-4665-a368-66bd25143c59" Nov 02 08:06:12.768715 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:12.768654 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rshcn" podUID="a55d1dbd-a295-4b42-a0fb-b16022237e8c" Nov 02 08:06:12.768715 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:12.768679 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:12.768820 ip-10-0-136-103 kubenswrapper[2570]: E1102 08:06:12.768756 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xs697" podUID="f4015660-eec1-47d8-8432-de3b74803f6a" Nov 02 08:06:12.835999 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:12.835958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:13.004245 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.004215 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:13.501509 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.501485 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeReady" Nov 02 08:06:13.501880 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.501617 2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 08:06:13.648247 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.648213 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jddzv"] Nov 02 08:06:13.673444 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.673416 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kck72"] Nov 02 08:06:13.673582 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.673533 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:13.689621 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.689593 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.689621 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.689599 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jddzv"] Nov 02 08:06:13.689810 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.689651 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kck72"] Nov 02 08:06:13.691772 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.691752 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l7nzr"] Nov 02 08:06:13.717656 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.717636 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.723660 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.723637 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 08:06:13.723746 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.723709 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t9rxp\"" Nov 02 08:06:13.739258 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.739241 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 08:06:13.739340 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.739257 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9tsd7\"" Nov 02 08:06:13.748904 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.748881 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l7nzr"] Nov 02 08:06:13.774432 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.774358 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 08:06:13.774432 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.774377 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 08:06:13.774432 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.774358 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 08:06:13.774432 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.774418 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 08:06:13.774432 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.774425 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 08:06:13.774730 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.774645 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 08:06:13.816298 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816271 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946e5a1e-bfdd-48e2-81a5-cb512473ef85-cert\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:13.816298 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816299 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2bm\" (UniqueName: \"kubernetes.io/projected/1294b06a-d192-4f92-bade-e90847b9b3d3-kube-api-access-7q2bm\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.816446 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816384 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d991975-b491-4d47-9345-1c021afcb155-data-volume\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.816446 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d991975-b491-4d47-9345-1c021afcb155-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.816446 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816429 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1294b06a-d192-4f92-bade-e90847b9b3d3-config-volume\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.816582 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d991975-b491-4d47-9345-1c021afcb155-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.816640 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816627 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prddx\" (UniqueName: \"kubernetes.io/projected/946e5a1e-bfdd-48e2-81a5-cb512473ef85-kube-api-access-prddx\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:13.816693 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816655 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294b06a-d192-4f92-bade-e90847b9b3d3-tmp-dir\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.816728 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1294b06a-d192-4f92-bade-e90847b9b3d3-metrics-tls\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.816728 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816722 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d991975-b491-4d47-9345-1c021afcb155-crio-socket\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.816801 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.816738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn862\" (UniqueName: \"kubernetes.io/projected/8d991975-b491-4d47-9345-1c021afcb155-kube-api-access-vn862\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.824295 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.824273 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 08:06:13.824379 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.824296 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9ztjt\"" Nov 02 08:06:13.917884 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.917853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d991975-b491-4d47-9345-1c021afcb155-crio-socket\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.917884 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.917883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn862\" (UniqueName: \"kubernetes.io/projected/8d991975-b491-4d47-9345-1c021afcb155-kube-api-access-vn862\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.918106 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.917906 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946e5a1e-bfdd-48e2-81a5-cb512473ef85-cert\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:13.918106 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.917922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2bm\" (UniqueName: \"kubernetes.io/projected/1294b06a-d192-4f92-bade-e90847b9b3d3-kube-api-access-7q2bm\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.918106 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d991975-b491-4d47-9345-1c021afcb155-data-volume\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.918106 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918091 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d991975-b491-4d47-9345-1c021afcb155-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.918343 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918105 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d991975-b491-4d47-9345-1c021afcb155-crio-socket\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.918343 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918148 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1294b06a-d192-4f92-bade-e90847b9b3d3-config-volume\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.918343 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d991975-b491-4d47-9345-1c021afcb155-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.918343 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918250 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prddx\" (UniqueName: \"kubernetes.io/projected/946e5a1e-bfdd-48e2-81a5-cb512473ef85-kube-api-access-prddx\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:13.918343 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294b06a-d192-4f92-bade-e90847b9b3d3-tmp-dir\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.918343 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918298 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1294b06a-d192-4f92-bade-e90847b9b3d3-metrics-tls\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.918678 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918649 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294b06a-d192-4f92-bade-e90847b9b3d3-tmp-dir\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.918844 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918825 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1294b06a-d192-4f92-bade-e90847b9b3d3-config-volume\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.918900 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918839 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d991975-b491-4d47-9345-1c021afcb155-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.918956 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.918936 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d991975-b491-4d47-9345-1c021afcb155-data-volume\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.921950 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.921926 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d991975-b491-4d47-9345-1c021afcb155-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.922036 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.921965 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1294b06a-d192-4f92-bade-e90847b9b3d3-metrics-tls\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.922077 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.922034 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946e5a1e-bfdd-48e2-81a5-cb512473ef85-cert\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:13.972336 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.972305 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn862\" (UniqueName: \"kubernetes.io/projected/8d991975-b491-4d47-9345-1c021afcb155-kube-api-access-vn862\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:13.972870 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.972855 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2bm\" (UniqueName: \"kubernetes.io/projected/1294b06a-d192-4f92-bade-e90847b9b3d3-kube-api-access-7q2bm\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:13.973280 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.973264 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prddx\" (UniqueName: \"kubernetes.io/projected/946e5a1e-bfdd-48e2-81a5-cb512473ef85-kube-api-access-prddx\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:13.995898 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:13.995873 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:14.047926 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.047844 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:14.076491 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.076453 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:14.364443 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.364348 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jddzv"] Nov 02 08:06:14.367599 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:14.367568 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946e5a1e_bfdd_48e2_81a5_cb512473ef85.slice/crio-70e067f8cdfb3cadac3af0a9ba448a7beb9a0fc9dba1c72274d6fdd9f45d23b3 WatchSource:0}: Error finding container 70e067f8cdfb3cadac3af0a9ba448a7beb9a0fc9dba1c72274d6fdd9f45d23b3: Status 404 returned error can't find the container with id 70e067f8cdfb3cadac3af0a9ba448a7beb9a0fc9dba1c72274d6fdd9f45d23b3 Nov 02 08:06:14.415630 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.415592 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kck72"] Nov 02 08:06:14.418252 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:14.418227 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d991975_b491_4d47_9345_1c021afcb155.slice/crio-d0b99317f8e68051a2ae8abfafcd9bd2d2c108d3150bf46821a327ee5c82fe5e WatchSource:0}: Error finding container d0b99317f8e68051a2ae8abfafcd9bd2d2c108d3150bf46821a327ee5c82fe5e: Status 404 returned error can't find the container with id d0b99317f8e68051a2ae8abfafcd9bd2d2c108d3150bf46821a327ee5c82fe5e Nov 02 08:06:14.465991 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.465969 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l7nzr"] Nov 02 08:06:14.468717 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:14.468689 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1294b06a_d192_4f92_bade_e90847b9b3d3.slice/crio-f692aa15c041ab5ad0a1336f92a892853b45278557eb26993ab47d0378af288f WatchSource:0}: Error finding container f692aa15c041ab5ad0a1336f92a892853b45278557eb26993ab47d0378af288f: Status 404 returned error can't find the container with id f692aa15c041ab5ad0a1336f92a892853b45278557eb26993ab47d0378af288f Nov 02 08:06:14.768665 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.768632 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:14.769074 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.768688 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:14.769074 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.768700 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:14.819339 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.819311 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 08:06:14.819339 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.819315 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 08:06:14.819339 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.819342 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 08:06:14.819550 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.819361 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 08:06:14.819550 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.819387 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nct\"" Nov 02 08:06:14.819550 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.819433 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghx8m\"" Nov 02 08:06:14.819550 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.819474 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4v7vr\"" Nov 02 08:06:14.998090 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.998060 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l7nzr" event={"ID":"1294b06a-d192-4f92-bade-e90847b9b3d3","Type":"ContainerStarted","Data":"f692aa15c041ab5ad0a1336f92a892853b45278557eb26993ab47d0378af288f"} Nov 02 08:06:14.999542 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.999518 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kck72" event={"ID":"8d991975-b491-4d47-9345-1c021afcb155","Type":"ContainerStarted","Data":"33ada1012323dbbc2d894764cd1f6392ae15a5189023f42dfdf2338ec787cd65"} Nov 02 08:06:14.999671 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:14.999561 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kck72" event={"ID":"8d991975-b491-4d47-9345-1c021afcb155","Type":"ContainerStarted","Data":"d0b99317f8e68051a2ae8abfafcd9bd2d2c108d3150bf46821a327ee5c82fe5e"} Nov 02 08:06:15.000665 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:15.000646 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jddzv" event={"ID":"946e5a1e-bfdd-48e2-81a5-cb512473ef85","Type":"ContainerStarted","Data":"70e067f8cdfb3cadac3af0a9ba448a7beb9a0fc9dba1c72274d6fdd9f45d23b3"} Nov 02 08:06:15.842345 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:15.842323 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ndqtq_d54dbca4-a028-4ec4-8aae-6edd83aaff7b/dns-node-resolver/0.log" Nov 02 08:06:15.899404 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:15.899372 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:16.005889 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:16.005798 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kck72" event={"ID":"8d991975-b491-4d47-9345-1c021afcb155","Type":"ContainerStarted","Data":"f53523bdb6360fb3276e64acb25150c362769126b966caaf75ead3083bc8df03"} Nov 02 08:06:16.642755 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:16.642721 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8v9wz_944dd579-cc50-401c-a32f-04055024d5f8/node-ca/0.log" Nov 02 08:06:16.644674 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:16.644601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:16.851877 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:16.851836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:16.988562 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:16.988532 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6"] Nov 02 08:06:16.991620 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:16.991596 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.042778 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.042754 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 08:06:17.063469 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.063441 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:17.090629 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.090598 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 08:06:17.090785 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.090694 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sggrc\"" Nov 02 08:06:17.090785 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.090740 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 08:06:17.090894 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.090827 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 08:06:17.090951 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.090924 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 08:06:17.092223 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.092167 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6"] Nov 02 08:06:17.147640 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.147611 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.147786 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.147683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.147786 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.147716 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwq8d\" (UniqueName: \"kubernetes.io/projected/68679646-df21-499f-b5dd-c5b6ceb12314-kube-api-access-kwq8d\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.147786 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.147783 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68679646-df21-499f-b5dd-c5b6ceb12314-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.248222 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.248188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68679646-df21-499f-b5dd-c5b6ceb12314-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.248416 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.248248 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.248416 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.248301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.248416 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.248334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwq8d\" (UniqueName: \"kubernetes.io/projected/68679646-df21-499f-b5dd-c5b6ceb12314-kube-api-access-kwq8d\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.249151 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.248964 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68679646-df21-499f-b5dd-c5b6ceb12314-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.251636 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.251574 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.262298 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.262276 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.305589 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.305545 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwq8d\" (UniqueName: \"kubernetes.io/projected/68679646-df21-499f-b5dd-c5b6ceb12314-kube-api-access-kwq8d\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.310195 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.310153 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:17.684094 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:17.684051 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6"] Nov 02 08:06:17.695398 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:17.695366 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68679646_df21_499f_b5dd_c5b6ceb12314.slice/crio-674d2b682e37c20c3919821046c794399acd8984e5484b759f1ed0a9b15e1eeb WatchSource:0}: Error finding container 674d2b682e37c20c3919821046c794399acd8984e5484b759f1ed0a9b15e1eeb: Status 404 returned error can't find the container with id 674d2b682e37c20c3919821046c794399acd8984e5484b759f1ed0a9b15e1eeb Nov 02 08:06:18.013941 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.013910 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l7nzr" event={"ID":"1294b06a-d192-4f92-bade-e90847b9b3d3","Type":"ContainerStarted","Data":"d773280950f2978a0c0829c0684b14d2bea4e99e56466a312c1a244ba008d801"} Nov 02 08:06:18.014359 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.014001 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l7nzr" event={"ID":"1294b06a-d192-4f92-bade-e90847b9b3d3","Type":"ContainerStarted","Data":"cf909de0e9ad56466154d23f2ec7e0e5ce75ccf511fb40d0ff2edf4c69e80d64"} Nov 02 08:06:18.014359 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.014029 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:18.016309 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.016284 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kck72" event={"ID":"8d991975-b491-4d47-9345-1c021afcb155","Type":"ContainerStarted","Data":"53ef7d712e4c8a3a0cab0c0a989c44d8815c4a3dba16905f846fa93ec6c029ab"} Nov 02 08:06:18.017410 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.017383 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" event={"ID":"68679646-df21-499f-b5dd-c5b6ceb12314","Type":"ContainerStarted","Data":"674d2b682e37c20c3919821046c794399acd8984e5484b759f1ed0a9b15e1eeb"} Nov 02 08:06:18.018896 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.018874 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jddzv" event={"ID":"946e5a1e-bfdd-48e2-81a5-cb512473ef85","Type":"ContainerStarted","Data":"aa2ce5eec0c9b467471630853439eff3c024c722a60652c0fbdc831004bb4dde"} Nov 02 08:06:18.122144 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.122082 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-l7nzr" podStartSLOduration=2.683896276 podStartE2EDuration="5.122068953s" totalImagesPullingTime="2.438172677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:18.121248692 +0000 UTC m=+49.868242359" watchObservedRunningTime="2026-11-02 08:06:18.122068953 +0000 UTC m=+49.869062620" Nov 02 08:06:18.229724 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.229601 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jddzv" podStartSLOduration=1.638000671 podStartE2EDuration="5.229588251s" totalImagesPullingTime="3.59158758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:18.228484147 +0000 UTC m=+49.975477814" watchObservedRunningTime="2026-11-02 08:06:18.229588251 +0000 UTC m=+49.976581919" Nov 02 08:06:18.257793 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.257760 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:18.338404 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.338346 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kck72" podStartSLOduration=1.95689686 podStartE2EDuration="5.338328027s" totalImagesPullingTime="3.381431167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:06:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:18.337448926 +0000 UTC m=+50.084442596" watchObservedRunningTime="2026-11-02 08:06:18.338328027 +0000 UTC m=+50.085321696" Nov 02 08:06:18.410981 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:18.410942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:19.003190 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.003142 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bcd5f847b-zrw55"] Nov 02 08:06:19.005187 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.005157 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.055668 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.055640 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 08:06:19.103721 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.103252 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 08:06:19.103721 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.103257 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 08:06:19.103721 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.103289 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wxtll\"" Nov 02 08:06:19.103721 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.103259 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 08:06:19.103721 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.103328 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 08:06:19.103721 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.103330 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 08:06:19.103721 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.103627 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 08:06:19.163125 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.163093 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.163309 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.163137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7k9\" (UniqueName: \"kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.163309 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.163199 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.163309 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.163233 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.163309 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.163267 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.163309 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.163304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.163564 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.163321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.163564 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.163344 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.199087 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.199052 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 08:06:19.203698 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.203333 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcd5f847b-zrw55"] Nov 02 08:06:19.263915 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.263830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.263915 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.263883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.263915 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.263904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.264231 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.263932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.264231 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.263961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.264231 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.264040 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.264231 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.264062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7k9\" (UniqueName: \"kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.264231 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.264088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.264532 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.264502 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.264793 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.264745 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.264793 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.264763 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.264989 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.264924 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.265110 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.265089 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.267198 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.266844 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.267291 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.267271 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.319416 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.319378 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7k9\" (UniqueName: \"kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.616101 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.616069 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:19.975594 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:19.975571 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcd5f847b-zrw55"] Nov 02 08:06:19.978158 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:19.978132 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd4a3e6_5543_48a2_8cdf_faa3b33ca11c.slice/crio-45f95867ea5f3f86697e78995bc373467e88e894eb245f262be2e65deaf2c9a3 WatchSource:0}: Error finding container 45f95867ea5f3f86697e78995bc373467e88e894eb245f262be2e65deaf2c9a3: Status 404 returned error can't find the container with id 45f95867ea5f3f86697e78995bc373467e88e894eb245f262be2e65deaf2c9a3 Nov 02 08:06:20.026805 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:20.026765 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" event={"ID":"68679646-df21-499f-b5dd-c5b6ceb12314","Type":"ContainerStarted","Data":"42b9d8279112bcfdf7abe1336213cc91326a80071a3a281dca23d7934c166f69"} Nov 02 08:06:20.026989 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:20.026824 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" event={"ID":"68679646-df21-499f-b5dd-c5b6ceb12314","Type":"ContainerStarted","Data":"b0e1414bc97c55a2940a34e41b875071844ef700f4375c7bdb3c5a5e65bcac45"} Nov 02 08:06:20.027955 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:20.027931 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5f847b-zrw55" event={"ID":"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c","Type":"ContainerStarted","Data":"45f95867ea5f3f86697e78995bc373467e88e894eb245f262be2e65deaf2c9a3"} Nov 02 08:06:20.137842 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:20.137785 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" podStartSLOduration=2.36093047 podStartE2EDuration="4.137770697s" totalImagesPullingTime="1.776840227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:20.137099526 +0000 UTC m=+51.884093217" watchObservedRunningTime="2026-11-02 08:06:20.137770697 +0000 UTC m=+51.884764364" Nov 02 08:06:20.667309 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:20.667273 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:20.869811 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:20.869765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:21.026418 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:21.026333 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:21.982949 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:21.982064 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:22.380955 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:22.380857 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:22.522932 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:22.522899 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:22.837239 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:22.837202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:23.037261 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.037168 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:23.202673 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.202643 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:23.364683 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.364644 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:23.445738 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.445705 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m89dq"] Nov 02 08:06:23.447627 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.447611 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.497848 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.497821 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 08:06:23.497848 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.497840 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lbg2j\"" Nov 02 08:06:23.529812 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.529781 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:23.564365 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.564338 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 08:06:23.564945 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.564933 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 08:06:23.600207 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.600163 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.600276 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.600226 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-textfile\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.600342 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.600323 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-metrics-client-ca\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.600397 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.600385 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-tls\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.600429 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.600411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-accelerators-collector-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.600470 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.600429 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-root\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.600539 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.600525 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-wtmp\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.600575 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.600549 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-sys\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.600575 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.600563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtv7l\" (UniqueName: \"kubernetes.io/projected/b5f83ac7-e0a5-47db-bf75-f72a32394313-kube-api-access-xtv7l\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.688272 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.688238 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:23.701504 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-tls\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701654 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701513 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-accelerators-collector-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701654 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-root\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701654 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701588 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-wtmp\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701654 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701616 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-sys\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701654 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtv7l\" (UniqueName: \"kubernetes.io/projected/b5f83ac7-e0a5-47db-bf75-f72a32394313-kube-api-access-xtv7l\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701894 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701681 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701894 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701688 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-root\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701894 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701715 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-sys\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701894 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701751 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-textfile\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701894 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701783 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-metrics-client-ca\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.701894 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.701795 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-wtmp\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.702263 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.702234 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-accelerators-collector-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.702345 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.702309 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-textfile\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.702408 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.702386 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-metrics-client-ca\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.704079 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.704031 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-tls\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.704278 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.704257 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.755656 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.755625 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtv7l\" (UniqueName: \"kubernetes.io/projected/b5f83ac7-e0a5-47db-bf75-f72a32394313-kube-api-access-xtv7l\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.759541 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.759502 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:23.769702 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:23.769674 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f83ac7_e0a5_47db_bf75_f72a32394313.slice/crio-b344834020a2e6cd711f70f0a085ebefc726305e85596a0d4faea0fe1982a6cc WatchSource:0}: Error finding container b344834020a2e6cd711f70f0a085ebefc726305e85596a0d4faea0fe1982a6cc: Status 404 returned error can't find the container with id b344834020a2e6cd711f70f0a085ebefc726305e85596a0d4faea0fe1982a6cc Nov 02 08:06:23.840026 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.839988 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:23.964985 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.964916 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:23.988406 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:23.988371 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:24.042021 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:24.041986 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5f847b-zrw55" event={"ID":"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c","Type":"ContainerStarted","Data":"207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07"} Nov 02 08:06:24.043275 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:24.043251 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m89dq" event={"ID":"b5f83ac7-e0a5-47db-bf75-f72a32394313","Type":"ContainerStarted","Data":"b344834020a2e6cd711f70f0a085ebefc726305e85596a0d4faea0fe1982a6cc"} Nov 02 08:06:24.194170 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:24.194119 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bcd5f847b-zrw55" podStartSLOduration=2.945347711 podStartE2EDuration="6.194105684s" totalImagesPullingTime="3.248757973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:24.193284044 +0000 UTC m=+55.940277713" watchObservedRunningTime="2026-11-02 08:06:24.194105684 +0000 UTC m=+55.941099352" Nov 02 08:06:24.765425 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:24.765366 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:24.963033 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:24.962992 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:25.402915 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.402882 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:25.547560 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.547480 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:25.644122 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.644095 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 08:06:25.646249 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.646233 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.675381 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.675347 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 08:06:25.721449 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.721418 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 08:06:25.721449 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.721442 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 08:06:25.721449 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.721456 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 08:06:25.721711 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.721499 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 08:06:25.721711 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.721506 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 08:06:25.721711 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.721508 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 08:06:25.722240 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.722202 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 08:06:25.722340 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.722310 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cxtc9\"" Nov 02 08:06:25.747755 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.747727 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:25.792287 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.792249 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 08:06:25.792423 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.792298 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 08:06:25.820991 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.820901 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.820991 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.820941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.820991 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.820979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821257 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821257 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821076 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821257 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821095 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821257 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821439 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821283 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821439 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821324 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821439 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821397 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821439 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821430 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821577 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821447 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzh8\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.821577 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.821470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.922675 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.922640 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.922675 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.922675 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.922865 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.922695 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.922865 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.922839 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.922965 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.922884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923044 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923093 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923139 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923106 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923231 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923148 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923231 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923340 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923340 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923290 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923340 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923340 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923328 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzh8\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.923740 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923717 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.924436 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.923910 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.925876 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.925849 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.926436 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.926354 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.926436 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.926360 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.926645 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.926618 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.926781 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.926760 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.926844 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.926813 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.926972 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.926954 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.927325 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.927310 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.927867 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.927846 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:25.977811 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:25.977784 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzh8\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:26.051082 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:26.051046 2570 generic.go:356] "Generic (PLEG): container finished" podID="b5f83ac7-e0a5-47db-bf75-f72a32394313" containerID="2906b926913e5094f13178c10646167e7534aca2ec6eae80f6b5923eea770e5b" exitCode=0 Nov 02 08:06:26.051230 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:26.051121 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m89dq" event={"ID":"b5f83ac7-e0a5-47db-bf75-f72a32394313","Type":"ContainerDied","Data":"2906b926913e5094f13178c10646167e7534aca2ec6eae80f6b5923eea770e5b"} Nov 02 08:06:26.262106 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:26.262074 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:26.721841 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:26.721820 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 08:06:26.723392 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:26.723358 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a387bc4_fe09_4921_9ff4_8199642acb38.slice/crio-8635ab5968e926289e2ff01e43c2988edbc62ef85663db5b03d7721ca42b8b5a WatchSource:0}: Error finding container 8635ab5968e926289e2ff01e43c2988edbc62ef85663db5b03d7721ca42b8b5a: Status 404 returned error can't find the container with id 8635ab5968e926289e2ff01e43c2988edbc62ef85663db5b03d7721ca42b8b5a Nov 02 08:06:27.056080 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:27.055986 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m89dq" event={"ID":"b5f83ac7-e0a5-47db-bf75-f72a32394313","Type":"ContainerStarted","Data":"ee694137d1b625030ea80b667f69c52fb9e6506822da297146bd6b591d4983cb"} Nov 02 08:06:27.056080 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:27.056062 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m89dq" event={"ID":"b5f83ac7-e0a5-47db-bf75-f72a32394313","Type":"ContainerStarted","Data":"1005b7ca08f7f759bd1d8f8d4de7310a865feb97023d328cc9e8f160486b7d36"} Nov 02 08:06:27.057195 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:27.057158 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerStarted","Data":"8635ab5968e926289e2ff01e43c2988edbc62ef85663db5b03d7721ca42b8b5a"} Nov 02 08:06:27.166213 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:27.166133 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m89dq" podStartSLOduration=2.589804699 podStartE2EDuration="4.166115306s" totalImagesPullingTime="1.499650298s" totalInitContainerRuntime="76.660309ms" isStatefulPod=true podCreationTimestamp="2026-11-02 08:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:27.165933748 +0000 UTC m=+58.912927430" watchObservedRunningTime="2026-11-02 08:06:27.166115306 +0000 UTC m=+58.913108976" Nov 02 08:06:27.427321 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:27.427280 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:27.645448 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:27.645409 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:27.791637 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:27.791558 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:27.950324 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:27.950288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:28.024324 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:28.024288 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:28.105285 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:28.105196 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:28.271468 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:28.271437 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:28.684783 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:28.683725 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:29.064805 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:29.064716 2570 generic.go:356] "Generic (PLEG): container finished" podID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerID="a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be" exitCode=0 Nov 02 08:06:29.065154 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:29.064800 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerDied","Data":"a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be"} Nov 02 08:06:29.616330 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:29.616301 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:29.616483 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:29.616354 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:29.621092 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:29.621073 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:30.071573 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:30.071548 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:31.073112 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:31.073084 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerStarted","Data":"325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f"} Nov 02 08:06:32.079809 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.079771 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerStarted","Data":"fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d"} Nov 02 08:06:32.080211 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.079832 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerStarted","Data":"f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b"} Nov 02 08:06:32.080211 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.079857 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerStarted","Data":"21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df"} Nov 02 08:06:32.080211 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.079879 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerStarted","Data":"6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607"} Nov 02 08:06:32.259225 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.259163 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:32.312891 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.312855 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 08:06:32.316523 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.316494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.367154 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.367067 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 08:06:32.367154 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.367076 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 08:06:32.367401 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.367256 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmnro8141ghht\"" Nov 02 08:06:32.380552 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.380521 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4qls\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.380711 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.380565 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.380711 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.380593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.380711 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.380665 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.380890 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.380754 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.380890 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.380829 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.380987 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.380912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.380987 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.380955 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.380987 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.380980 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381137 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381137 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381093 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381137 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381288 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381169 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381288 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381254 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381288 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381278 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381503 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381310 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381503 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381333 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381503 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381472 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381652 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381543 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.381709 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.381642 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.414798 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.414739 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 08:06:32.414982 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.414947 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 08:06:32.415620 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.415573 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 08:06:32.415745 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.415717 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 08:06:32.417232 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.417212 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 08:06:32.467160 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.467093 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 08:06:32.467160 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.467093 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xbrch\"" Nov 02 08:06:32.467160 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.467124 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 08:06:32.467160 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.467125 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 08:06:32.467436 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.467363 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 08:06:32.467436 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.467366 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 08:06:32.467507 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.467487 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 08:06:32.467544 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.467534 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 08:06:32.482086 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482281 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482281 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482133 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482281 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482157 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482281 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4qls\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482281 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482237 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482281 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482252 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482281 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482296 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482318 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482412 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482469 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482520 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482591 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.482649 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482635 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.483191 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482792 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.483191 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.482900 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.483902 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.483824 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.484010 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.483994 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.485454 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.485429 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.486610 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.486582 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.487913 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.487864 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.487980 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.487962 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.488014 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.487988 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.488855 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.488825 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.488993 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.488975 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.489060 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.488993 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.489132 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.489112 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.489165 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.489138 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.489616 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.489597 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.489677 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.489621 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.490989 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.490968 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.529244 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.529212 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 08:06:32.529457 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.529286 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 08:06:32.536369 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.536335 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.536963 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.536938 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.537869 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.537844 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4qls\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:32.641436 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:32.641322 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:33.061623 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.061593 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 08:06:33.065017 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:33.064987 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a4044b_18d1_494c_b50e_2b3d253efdf2.slice/crio-393a34c14ee09719bfdab0517a1b5cac495b200fd4fa3f6d8958e7e3ca608cad WatchSource:0}: Error finding container 393a34c14ee09719bfdab0517a1b5cac495b200fd4fa3f6d8958e7e3ca608cad: Status 404 returned error can't find the container with id 393a34c14ee09719bfdab0517a1b5cac495b200fd4fa3f6d8958e7e3ca608cad Nov 02 08:06:33.085237 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.085208 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerStarted","Data":"393a34c14ee09719bfdab0517a1b5cac495b200fd4fa3f6d8958e7e3ca608cad"} Nov 02 08:06:33.089007 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.088984 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerStarted","Data":"f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc"} Nov 02 08:06:33.208042 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.207992 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.332412047 podStartE2EDuration="8.20797779s" totalImagesPullingTime="4.863076626s" totalInitContainerRuntime="12.489117ms" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:33.206610452 +0000 UTC m=+64.953604139" watchObservedRunningTime="2026-11-02 08:06:33.20797779 +0000 UTC m=+64.954971458" Nov 02 08:06:33.794506 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.794476 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:33.794688 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.794517 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:33.843667 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.843638 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 08:06:33.843667 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.843669 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 08:06:33.847390 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.847365 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:33.847512 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.847373 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:33.897200 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.895760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:33.944890 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.944859 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 08:06:33.995461 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:33.995435 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 08:06:34.000137 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.000109 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:34.031137 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.031113 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4v7vr\"" Nov 02 08:06:34.033309 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.033157 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:34.039413 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.039393 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nct\"" Nov 02 08:06:34.040428 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.040405 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:34.107599 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.107500 2570 generic.go:356] "Generic (PLEG): container finished" podID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" exitCode=0 Nov 02 08:06:34.108256 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.107732 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerDied","Data":"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2"} Nov 02 08:06:34.335269 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.335142 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghx8m\"" Nov 02 08:06:34.337282 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.337253 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:34.416768 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.416696 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xs697"] Nov 02 08:06:34.419498 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:34.419466 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4015660_eec1_47d8_8432_de3b74803f6a.slice/crio-99bdf546e7515cd344458cdf6ca525ff4f25865be294bfd5b165ff2fd646d3b0 WatchSource:0}: Error finding container 99bdf546e7515cd344458cdf6ca525ff4f25865be294bfd5b165ff2fd646d3b0: Status 404 returned error can't find the container with id 99bdf546e7515cd344458cdf6ca525ff4f25865be294bfd5b165ff2fd646d3b0 Nov 02 08:06:34.465872 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.465839 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zjzfb"] Nov 02 08:06:34.473161 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:34.473134 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715adb78_caaa_4665_a368_66bd25143c59.slice/crio-82d1d3c0042721b0af053546d841ca6dafea3c07347eba6e08353ad36cde2b8b WatchSource:0}: Error finding container 82d1d3c0042721b0af053546d841ca6dafea3c07347eba6e08353ad36cde2b8b: Status 404 returned error can't find the container with id 82d1d3c0042721b0af053546d841ca6dafea3c07347eba6e08353ad36cde2b8b Nov 02 08:06:34.651235 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.651141 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:34.715226 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.715195 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rshcn"] Nov 02 08:06:34.717892 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:34.717861 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55d1dbd_a295_4b42_a0fb_b16022237e8c.slice/crio-2fcf0e7c7a1196d3df280b0c683f5b57d2f71dacd035cd2fe08bc1c5c78ee619 WatchSource:0}: Error finding container 2fcf0e7c7a1196d3df280b0c683f5b57d2f71dacd035cd2fe08bc1c5c78ee619: Status 404 returned error can't find the container with id 2fcf0e7c7a1196d3df280b0c683f5b57d2f71dacd035cd2fe08bc1c5c78ee619 Nov 02 08:06:34.810969 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.810938 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:34.977634 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:34.977598 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:35.112543 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.112510 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rshcn" event={"ID":"a55d1dbd-a295-4b42-a0fb-b16022237e8c","Type":"ContainerStarted","Data":"2fcf0e7c7a1196d3df280b0c683f5b57d2f71dacd035cd2fe08bc1c5c78ee619"} Nov 02 08:06:35.113827 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.113806 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zjzfb" event={"ID":"715adb78-caaa-4665-a368-66bd25143c59","Type":"ContainerStarted","Data":"82d1d3c0042721b0af053546d841ca6dafea3c07347eba6e08353ad36cde2b8b"} Nov 02 08:06:35.116491 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.115097 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xs697" event={"ID":"f4015660-eec1-47d8-8432-de3b74803f6a","Type":"ContainerStarted","Data":"99bdf546e7515cd344458cdf6ca525ff4f25865be294bfd5b165ff2fd646d3b0"} Nov 02 08:06:35.170619 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.170586 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dd89b74f-67l4d"] Nov 02 08:06:35.173441 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.173423 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.213019 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.212986 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.213165 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.213029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzpcr\" (UniqueName: \"kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.213165 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.213094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.213165 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.213137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.213165 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.213156 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.213309 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.213265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.213371 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.213354 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.213404 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.213393 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.228904 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.228826 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dd89b74f-67l4d"] Nov 02 08:06:35.314634 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.314595 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.314814 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.314645 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.314814 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.314669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.314814 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.314690 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.314814 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.314730 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.314814 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.314762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpcr\" (UniqueName: \"kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.314814 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.314793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.315114 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.314836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.315114 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.315087 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.315588 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.315559 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.315726 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.315592 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.315726 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.315713 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.316104 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.316085 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.317583 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.317567 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.317622 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.317593 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.370094 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.370059 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzpcr\" (UniqueName: \"kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.383771 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.383738 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:35.486383 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.486307 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:35.556529 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.556364 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:35.720766 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.720730 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:35.865312 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:35.865213 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dd89b74f-67l4d"] Nov 02 08:06:35.897042 ip-10-0-136-103 kubenswrapper[2570]: W1102 08:06:35.896983 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6a0a75_e046_4589_8711_4a0f1fd83a9d.slice/crio-70f2921a186f179e92aa7d509524b1302dbf6d0b7e3f26cba55a9e2b9644c307 WatchSource:0}: Error finding container 70f2921a186f179e92aa7d509524b1302dbf6d0b7e3f26cba55a9e2b9644c307: Status 404 returned error can't find the container with id 70f2921a186f179e92aa7d509524b1302dbf6d0b7e3f26cba55a9e2b9644c307 Nov 02 08:06:36.125063 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:36.123931 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd89b74f-67l4d" event={"ID":"3d6a0a75-e046-4589-8711-4a0f1fd83a9d","Type":"ContainerStarted","Data":"97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071"} Nov 02 08:06:36.125063 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:36.123990 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd89b74f-67l4d" event={"ID":"3d6a0a75-e046-4589-8711-4a0f1fd83a9d","Type":"ContainerStarted","Data":"70f2921a186f179e92aa7d509524b1302dbf6d0b7e3f26cba55a9e2b9644c307"} Nov 02 08:06:36.242149 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:36.242091 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5dd89b74f-67l4d" podStartSLOduration=1.242076347 podStartE2EDuration="1.242076347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:36.234905758 +0000 UTC m=+67.981899425" watchObservedRunningTime="2026-11-02 08:06:36.242076347 +0000 UTC m=+67.989070015" Nov 02 08:06:37.132314 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:37.132232 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zjzfb" event={"ID":"715adb78-caaa-4665-a368-66bd25143c59","Type":"ContainerStarted","Data":"0df6d64525918bc754a8e8b1355a30277cc3e1787117c6d4e9d7804fdc8560b5"} Nov 02 08:06:37.132865 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:37.132323 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zjzfb" event={"ID":"715adb78-caaa-4665-a368-66bd25143c59","Type":"ContainerStarted","Data":"5b445d8348851f9eda9d608c16ff3ad256513b8cb4e455bdbc123c8e5f2c58dd"} Nov 02 08:06:37.221801 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:37.221765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:37.246264 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:37.246209 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zjzfb" podStartSLOduration=66.131766139 podStartE2EDuration="1m8.24616822s" totalImagesPullingTime="2.114402081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:37.245485825 +0000 UTC m=+68.992479493" watchObservedRunningTime="2026-11-02 08:06:37.24616822 +0000 UTC m=+68.993161891" Nov 02 08:06:37.417471 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:37.417402 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:37.590408 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:37.590362 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:39.146431 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:39.146402 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerStarted","Data":"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568"} Nov 02 08:06:39.638058 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:39.638026 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:39.793975 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:39.793943 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:40.154673 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:40.154631 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rshcn" event={"ID":"a55d1dbd-a295-4b42-a0fb-b16022237e8c","Type":"ContainerStarted","Data":"0ad933f6a979530dfb0ffc1c436e0fc4b298830089b8af16cbab43c1f010c5c9"} Nov 02 08:06:40.155154 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:40.154904 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:40.157387 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:40.157359 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerStarted","Data":"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7"} Nov 02 08:06:40.264055 ip-10-0-136-103 kubenswrapper[2570]: I1102 08:06:40.263995 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rshcn" podStartSLOduration=67.2895621 podStartE2EDuration="1m12.263974605s" totalImagesPullingTime="4.974412505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:40.262852849 +0000 UTC m=+72.009846536" watchObservedRunningTime="2026-11-02 08:06:40.263974605 +0000 UTC m=+72.010968272" Nov 02 08:06:41.372454 ip-10-0-136-103 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 08:06:41.390951 ip-10-0-136-103 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 08:06:41.391142 ip-10-0-136-103 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 08:06:41.391223 ip-10-0-136-103 systemd[1]: kubelet.service: Consumed 4.372s CPU time. Nov 02 08:06:41.401768 ip-10-0-136-103 systemd[1]: Starting Kubernetes Kubelet... Nov 02 08:06:41.482820 ip-10-0-136-103 kubenswrapper[6160]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:06:41.482820 ip-10-0-136-103 kubenswrapper[6160]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 08:06:41.482820 ip-10-0-136-103 kubenswrapper[6160]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:06:41.482820 ip-10-0-136-103 kubenswrapper[6160]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 08:06:41.485199 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485086 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:06:41.485199 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485196 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:06:41.485199 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485201 6160 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:06:41.485199 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485205 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:06:41.485199 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485208 6160 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485211 6160 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485214 6160 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485217 6160 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485226 6160 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485229 6160 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485232 6160 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485235 6160 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485239 6160 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485243 6160 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485247 6160 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485251 6160 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485256 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485261 6160 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485265 6160 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485270 6160 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485274 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485279 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485283 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485287 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:06:41.485423 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485292 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485297 6160 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485301 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485305 6160 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485309 6160 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485313 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485320 6160 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485327 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485332 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485335 6160 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485341 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485345 6160 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485350 6160 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485354 6160 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485357 6160 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485361 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485364 6160 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485369 6160 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485375 6160 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:06:41.485996 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485380 6160 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485385 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485389 6160 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485395 6160 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485399 6160 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485403 6160 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485408 6160 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485419 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485422 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485425 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485428 6160 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485431 6160 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485433 6160 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485436 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485439 6160 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485443 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485446 6160 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485448 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485451 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485454 6160 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:06:41.486545 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485456 6160 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485459 6160 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485461 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485464 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485466 6160 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485469 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485471 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485475 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485477 6160 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485481 6160 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485484 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485486 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485489 6160 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485492 6160 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485497 6160 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485501 6160 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485505 6160 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485509 6160 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485513 6160 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485515 6160 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:06:41.487136 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485518 6160 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485521 6160 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485523 6160 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485526 6160 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485529 6160 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485532 6160 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485535 6160 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485537 6160 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485540 6160 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485543 6160 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485546 6160 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485549 6160 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485551 6160 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485556 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485560 6160 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485564 6160 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485567 6160 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485570 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485573 6160 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:06:41.487717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485578 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485581 6160 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485583 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.485587 6160 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486527 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486535 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486539 6160 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486542 6160 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486545 6160 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486549 6160 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486554 6160 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486558 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486563 6160 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486568 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486572 6160 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486577 6160 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486581 6160 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486587 6160 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486591 6160 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:06:41.488344 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486595 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486598 6160 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486601 6160 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486603 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486607 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486609 6160 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486612 6160 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486614 6160 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486617 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486620 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486622 6160 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486626 6160 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486630 6160 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486634 6160 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486639 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486644 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486650 6160 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486656 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486663 6160 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:06:41.488882 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486671 6160 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486675 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486680 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486684 6160 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486688 6160 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486692 6160 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486696 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486700 6160 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486704 6160 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486708 6160 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486712 6160 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486716 6160 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486719 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486727 6160 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486732 6160 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486734 6160 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486737 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486740 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486743 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486745 6160 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:06:41.489519 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486749 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486751 6160 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486754 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486756 6160 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486759 6160 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486761 6160 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486764 6160 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486767 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486770 6160 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486772 6160 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486775 6160 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486778 6160 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486781 6160 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486783 6160 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486786 6160 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486788 6160 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486790 6160 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486794 6160 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486798 6160 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486802 6160 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:06:41.490119 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486807 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486810 6160 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486812 6160 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486815 6160 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486818 6160 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486822 6160 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486825 6160 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486829 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486833 6160 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486838 6160 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486843 6160 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486847 6160 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486852 6160 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486856 6160 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486861 6160 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486865 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486870 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486874 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486878 6160 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486880 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:06:41.490753 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486883 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486886 6160 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486888 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486890 6160 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486894 6160 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486897 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486899 6160 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486902 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486904 6160 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486907 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486910 6160 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.486912 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487474 6160 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487483 6160 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487488 6160 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487491 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487494 6160 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487499 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487502 6160 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:06:41.491377 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487504 6160 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487507 6160 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487510 6160 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487513 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487518 6160 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487522 6160 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487527 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487531 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487534 6160 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487537 6160 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487539 6160 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487542 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487544 6160 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487547 6160 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487549 6160 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487552 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487554 6160 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487558 6160 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487560 6160 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:06:41.491946 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487564 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487566 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487569 6160 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487571 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487574 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487576 6160 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487578 6160 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487581 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487583 6160 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487586 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487592 6160 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487597 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487602 6160 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487607 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487610 6160 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487612 6160 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487615 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487617 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487620 6160 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487623 6160 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:06:41.492764 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487626 6160 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487628 6160 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487631 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487634 6160 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487638 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487642 6160 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487645 6160 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487648 6160 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487650 6160 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487653 6160 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487657 6160 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487661 6160 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487666 6160 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487672 6160 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487677 6160 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487682 6160 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487686 6160 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487691 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487695 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487699 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:06:41.493634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487701 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487704 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487709 6160 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487715 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487720 6160 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487723 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487725 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487728 6160 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487731 6160 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487733 6160 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487736 6160 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487739 6160 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487741 6160 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487744 6160 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487746 6160 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487749 6160 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487751 6160 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487754 6160 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487756 6160 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487759 6160 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:06:41.494549 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487761 6160 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487764 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487766 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487769 6160 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487772 6160 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487775 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487778 6160 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487780 6160 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487782 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487787 6160 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487790 6160 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487793 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487795 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487798 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487801 6160 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487804 6160 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487808 6160 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487811 6160 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487813 6160 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:06:41.495368 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.487815 6160 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487935 6160 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487944 6160 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487951 6160 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487956 6160 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487960 6160 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487964 6160 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487968 6160 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487972 6160 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487975 6160 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487978 6160 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487982 6160 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487985 6160 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487988 6160 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487991 6160 flags.go:64] FLAG: --cgroup-root="" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487994 6160 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487997 6160 flags.go:64] FLAG: --client-ca-file="" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.487999 6160 flags.go:64] FLAG: --cloud-provider="external" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488003 6160 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488007 6160 flags.go:64] FLAG: --cluster-domain="" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488010 6160 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488014 6160 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488019 6160 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488025 6160 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 08:06:41.496099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488031 6160 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488036 6160 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488041 6160 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488049 6160 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488053 6160 flags.go:64] FLAG: --contention-profiling="false" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488058 6160 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488062 6160 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488065 6160 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488068 6160 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488072 6160 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488075 6160 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488078 6160 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488081 6160 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488084 6160 flags.go:64] FLAG: --enable-server="true" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488087 6160 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488092 6160 flags.go:64] FLAG: --event-burst="100" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488095 6160 flags.go:64] FLAG: --event-qps="50" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488098 6160 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488102 6160 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488104 6160 flags.go:64] FLAG: --eviction-hard="" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488109 6160 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488111 6160 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488114 6160 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488117 6160 flags.go:64] FLAG: --eviction-soft="" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488120 6160 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 08:06:41.496833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488123 6160 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488126 6160 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488129 6160 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488133 6160 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488137 6160 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488141 6160 flags.go:64] FLAG: --feature-gates="" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488146 6160 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488151 6160 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488156 6160 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488162 6160 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488169 6160 flags.go:64] FLAG: --healthz-port="10248" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488190 6160 flags.go:64] FLAG: --help="false" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488194 6160 flags.go:64] FLAG: --hostname-override="ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488200 6160 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488204 6160 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488209 6160 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488214 6160 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488219 6160 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488223 6160 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488228 6160 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488233 6160 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488236 6160 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488239 6160 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 08:06:41.497727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488242 6160 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488245 6160 flags.go:64] FLAG: --kube-reserved="" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488248 6160 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488250 6160 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488253 6160 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488256 6160 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488259 6160 flags.go:64] FLAG: --lock-file="" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488262 6160 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488264 6160 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488267 6160 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488273 6160 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488276 6160 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488280 6160 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488283 6160 flags.go:64] FLAG: --logging-format="text" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488286 6160 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488289 6160 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488291 6160 flags.go:64] FLAG: --manifest-url="" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488294 6160 flags.go:64] FLAG: --manifest-url-header="" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488298 6160 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488304 6160 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488310 6160 flags.go:64] FLAG: --max-pods="110" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488314 6160 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488320 6160 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488324 6160 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488328 6160 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 08:06:41.498389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488333 6160 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488337 6160 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488341 6160 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488351 6160 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488357 6160 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488360 6160 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488363 6160 flags.go:64] FLAG: --pod-cidr="" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488366 6160 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488368 6160 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488371 6160 flags.go:64] FLAG: --pods-per-core="0" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488376 6160 flags.go:64] FLAG: --port="10250" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488379 6160 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488382 6160 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01b4be2ff752e38a8" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488385 6160 flags.go:64] FLAG: --qos-reserved="" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488388 6160 flags.go:64] FLAG: --read-only-port="10255" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488391 6160 flags.go:64] FLAG: --register-node="true" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488394 6160 flags.go:64] FLAG: --register-with-taints="" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488398 6160 flags.go:64] FLAG: --registry-burst="10" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488401 6160 flags.go:64] FLAG: --registry-qps="5" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488404 6160 flags.go:64] FLAG: --reserved-cpus="" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488407 6160 flags.go:64] FLAG: --reserved-memory="" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488411 6160 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488415 6160 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488417 6160 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488421 6160 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488423 6160 flags.go:64] FLAG: --runonce="false" Nov 02 08:06:41.499083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488427 6160 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488430 6160 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488433 6160 flags.go:64] FLAG: --seccomp-default="false" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488436 6160 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488439 6160 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488442 6160 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488445 6160 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488448 6160 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488451 6160 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488454 6160 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488457 6160 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488460 6160 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488463 6160 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488466 6160 flags.go:64] FLAG: --system-cgroups="" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488470 6160 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488479 6160 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488485 6160 flags.go:64] FLAG: --tls-cert-file="" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488488 6160 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488492 6160 flags.go:64] FLAG: --tls-min-version="" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488497 6160 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488501 6160 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488506 6160 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488511 6160 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488516 6160 flags.go:64] FLAG: --v="2" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488522 6160 flags.go:64] FLAG: --version="false" Nov 02 08:06:41.499851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488528 6160 flags.go:64] FLAG: --vmodule="" Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488533 6160 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.488539 6160 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488693 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488698 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488701 6160 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488704 6160 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488708 6160 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488710 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488713 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488716 6160 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488718 6160 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488722 6160 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488724 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488727 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488730 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488732 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488735 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488737 6160 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488740 6160 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:06:41.500594 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488744 6160 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488749 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488754 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488760 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488763 6160 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488766 6160 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488769 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488771 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488774 6160 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488776 6160 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488779 6160 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488782 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488784 6160 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488787 6160 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488790 6160 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488793 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488796 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488799 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488801 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:06:41.501165 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488807 6160 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488810 6160 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488812 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488815 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488817 6160 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488822 6160 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488825 6160 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488828 6160 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488831 6160 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488834 6160 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488837 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488841 6160 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488845 6160 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488849 6160 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488853 6160 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488858 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488864 6160 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488869 6160 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488874 6160 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:06:41.501835 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488878 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488884 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488888 6160 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488893 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488897 6160 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488903 6160 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488907 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488912 6160 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488918 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488924 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488930 6160 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488934 6160 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488940 6160 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488947 6160 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488952 6160 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488956 6160 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488960 6160 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488963 6160 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488965 6160 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488969 6160 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:06:41.502538 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488971 6160 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488974 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488976 6160 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488979 6160 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488981 6160 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488984 6160 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488988 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488990 6160 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488993 6160 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.488997 6160 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489000 6160 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489002 6160 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489005 6160 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489007 6160 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489010 6160 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489013 6160 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489015 6160 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489017 6160 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489021 6160 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489026 6160 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:06:41.503201 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489030 6160 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489034 6160 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489038 6160 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489043 6160 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489047 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489052 6160 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489056 6160 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489059 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489063 6160 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489067 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489071 6160 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.489111 6160 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489155 6160 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489160 6160 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489163 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:06:41.504060 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489166 6160 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489169 6160 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489189 6160 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489193 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489197 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489201 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489204 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489208 6160 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489210 6160 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489213 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489216 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489218 6160 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489223 6160 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489229 6160 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489233 6160 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489239 6160 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489243 6160 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489247 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489251 6160 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:06:41.504740 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489255 6160 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489259 6160 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489263 6160 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489267 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489272 6160 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489276 6160 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489279 6160 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489282 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489285 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489287 6160 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489290 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489292 6160 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489295 6160 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489297 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489301 6160 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489305 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489308 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489311 6160 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489314 6160 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:06:41.505406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489317 6160 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489322 6160 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489325 6160 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489328 6160 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489330 6160 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489333 6160 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489336 6160 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489338 6160 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489341 6160 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489343 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489346 6160 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489348 6160 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489351 6160 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489353 6160 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489356 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489358 6160 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489362 6160 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489364 6160 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489367 6160 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489369 6160 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:06:41.506070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489372 6160 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489375 6160 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489377 6160 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489381 6160 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489385 6160 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489389 6160 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489393 6160 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489397 6160 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489401 6160 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489406 6160 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489410 6160 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489422 6160 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489427 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489434 6160 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489438 6160 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489441 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489443 6160 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489446 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489449 6160 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489451 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:06:41.506759 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489454 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489457 6160 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489462 6160 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489466 6160 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489471 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489475 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489479 6160 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489482 6160 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489487 6160 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489490 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489493 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489496 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489499 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489501 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489503 6160 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489507 6160 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489510 6160 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489512 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489515 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489517 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:06:41.507430 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489520 6160 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489523 6160 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489525 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489528 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.489531 6160 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.496524 6160 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.496541 6160 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.496559 6160 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.496564 6160 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496618 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496622 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496626 6160 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496629 6160 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496632 6160 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496635 6160 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496637 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496640 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496642 6160 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496645 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:06:41.508006 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496647 6160 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496651 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496654 6160 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496656 6160 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496659 6160 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496667 6160 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496670 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496672 6160 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496675 6160 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496678 6160 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496680 6160 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496683 6160 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496685 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496688 6160 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496690 6160 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496693 6160 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496695 6160 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496698 6160 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496701 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496703 6160 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:06:41.508576 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496706 6160 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496708 6160 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496711 6160 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496713 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496716 6160 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496718 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496721 6160 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496723 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496726 6160 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496728 6160 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496730 6160 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496733 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496735 6160 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496738 6160 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496741 6160 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496743 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496747 6160 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496751 6160 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496756 6160 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:06:41.509265 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496759 6160 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496761 6160 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496764 6160 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496766 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496769 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496772 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496774 6160 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496777 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496779 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496782 6160 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496784 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496787 6160 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496790 6160 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496792 6160 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496795 6160 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496797 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496800 6160 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496804 6160 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496808 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496812 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:06:41.509791 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496816 6160 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496819 6160 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496824 6160 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496827 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496830 6160 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496832 6160 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496835 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496838 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496842 6160 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496844 6160 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496849 6160 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496852 6160 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496855 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496858 6160 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496861 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496863 6160 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496866 6160 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496868 6160 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496871 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:06:41.510388 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496873 6160 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496876 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496878 6160 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496881 6160 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496883 6160 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496886 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496889 6160 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496891 6160 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496894 6160 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496896 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496899 6160 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496902 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496904 6160 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496906 6160 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496909 6160 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496911 6160 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496914 6160 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:06:41.510941 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.496916 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.496945 6160 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497125 6160 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497132 6160 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497138 6160 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497143 6160 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497147 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497154 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497158 6160 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497162 6160 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497167 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497195 6160 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497200 6160 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497204 6160 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497209 6160 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497214 6160 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:06:41.511496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497218 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497222 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497227 6160 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497231 6160 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497236 6160 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497239 6160 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497242 6160 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497245 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497248 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497250 6160 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497253 6160 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497255 6160 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497258 6160 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497260 6160 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497263 6160 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497266 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497268 6160 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497271 6160 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497273 6160 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497275 6160 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:06:41.511969 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497278 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497280 6160 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497282 6160 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497285 6160 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497289 6160 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497292 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497294 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497297 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497299 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497302 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497304 6160 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497307 6160 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497309 6160 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497312 6160 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497314 6160 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497317 6160 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497321 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497323 6160 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497326 6160 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497328 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:06:41.512600 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497331 6160 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497334 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497336 6160 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497338 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497341 6160 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497343 6160 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497346 6160 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497348 6160 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497351 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497353 6160 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497356 6160 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497358 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497361 6160 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497363 6160 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497365 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497368 6160 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497370 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497373 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497376 6160 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497378 6160 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:06:41.513456 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497380 6160 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497383 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497385 6160 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497388 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497392 6160 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497396 6160 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497400 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497405 6160 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497410 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497423 6160 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497426 6160 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497428 6160 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497431 6160 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497434 6160 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497437 6160 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497439 6160 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497442 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497445 6160 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497448 6160 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:06:41.514362 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497450 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497453 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497457 6160 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497461 6160 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497465 6160 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497469 6160 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497472 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497475 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497478 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497480 6160 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497483 6160 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497487 6160 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497490 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.497520 6160 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497566 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 08:06:41.514914 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497573 6160 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497578 6160 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497583 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497587 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497591 6160 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497596 6160 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497601 6160 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497606 6160 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497612 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497616 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497620 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497622 6160 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497625 6160 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497628 6160 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497631 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497633 6160 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497636 6160 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497638 6160 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497641 6160 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 08:06:41.515551 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497643 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497646 6160 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497648 6160 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497651 6160 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497653 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497656 6160 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497658 6160 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497661 6160 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497663 6160 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497666 6160 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497669 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497672 6160 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497674 6160 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497677 6160 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497679 6160 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497681 6160 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497684 6160 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497686 6160 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497689 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497693 6160 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 08:06:41.516146 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497695 6160 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497698 6160 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497700 6160 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497703 6160 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497705 6160 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497708 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497710 6160 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497714 6160 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497718 6160 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497722 6160 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497726 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497730 6160 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497734 6160 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497738 6160 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497742 6160 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497746 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497750 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497754 6160 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497758 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497761 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 08:06:41.516788 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497764 6160 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497767 6160 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497769 6160 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497773 6160 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497777 6160 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497781 6160 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497786 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497790 6160 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497793 6160 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497796 6160 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497802 6160 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497805 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497808 6160 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497810 6160 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497813 6160 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497816 6160 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497819 6160 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497821 6160 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497824 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 08:06:41.517385 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497826 6160 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497828 6160 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497831 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497833 6160 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497836 6160 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497838 6160 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497840 6160 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497843 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497845 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497848 6160 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497850 6160 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497853 6160 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497855 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497858 6160 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497860 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497862 6160 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497865 6160 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497868 6160 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497872 6160 feature_gate.go:386] unrecognized feature gate: Example Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497874 6160 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 08:06:41.517927 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497876 6160 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497879 6160 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497881 6160 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497885 6160 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497888 6160 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497890 6160 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:06:41.497894 6160 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.497967 6160 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.499098 6160 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.499166 6160 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.499406 6160 server.go:1014] "Starting client certificate rotation" Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.499561 6160 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.499633 6160 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 08:00:28 +0000 UTC" deadline="2028-06-24 18:57:37.128660695 +0000 UTC" Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.499672 6160 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14410h50m55.628992357s" Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.499972 6160 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.501307 6160 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 08:06:41.518496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.502588 6160 log.go:25] "Validated CRI v1 runtime API" Nov 02 08:06:41.518987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.505012 6160 log.go:25] "Validated CRI v1 image API" Nov 02 08:06:41.518987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.505928 6160 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 08:06:41.518987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.508276 6160 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e1628e1a-9bf3-4d60-b390-605570e7a227:/dev/nvme0n1p3 e7888639-533f-44be-b6ce-c17d75aca700:/dev/nvme0n1p4] Nov 02 08:06:41.519116 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.508296 6160 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fcf0e7c7a1196d3df280b0c683f5b57d2f71dacd035cd2fe08bc1c5c78ee619/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fcf0e7c7a1196d3df280b0c683f5b57d2f71dacd035cd2fe08bc1c5c78ee619/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/393a34c14ee09719bfdab0517a1b5cac495b200fd4fa3f6d8958e7e3ca608cad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/393a34c14ee09719bfdab0517a1b5cac495b200fd4fa3f6d8958e7e3ca608cad/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45f95867ea5f3f86697e78995bc373467e88e894eb245f262be2e65deaf2c9a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45f95867ea5f3f86697e78995bc373467e88e894eb245f262be2e65deaf2c9a3/userdata/shm major:0 minor:268 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/646adad95a53d1ac07326d5ca32a8c5f169d7220beab88980809d0bd77845583/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/646adad95a53d1ac07326d5ca32a8c5f169d7220beab88980809d0bd77845583/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/674d2b682e37c20c3919821046c794399acd8984e5484b759f1ed0a9b15e1eeb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/674d2b682e37c20c3919821046c794399acd8984e5484b759f1ed0a9b15e1eeb/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70e067f8cdfb3cadac3af0a9ba448a7beb9a0fc9dba1c72274d6fdd9f45d23b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70e067f8cdfb3cadac3af0a9ba448a7beb9a0fc9dba1c72274d6fdd9f45d23b3/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70f2921a186f179e92aa7d509524b1302dbf6d0b7e3f26cba55a9e2b9644c307/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70f2921a186f179e92aa7d509524b1302dbf6d0b7e3f26cba55a9e2b9644c307/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82d1d3c0042721b0af053546d841ca6dafea3c07347eba6e08353ad36cde2b8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82d1d3c0042721b0af053546d841ca6dafea3c07347eba6e08353ad36cde2b8b/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82f719c36611a6097abc16cf244a1295a4b693ef3dbd19d6211583308d4c0757/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82f719c36611a6097abc16cf244a1295a4b693ef3dbd19d6211583308d4c0757/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8635ab5968e926289e2ff01e43c2988edbc62ef85663db5b03d7721ca42b8b5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8635ab5968e926289e2ff01e43c2988edbc62ef85663db5b03d7721ca42b8b5a/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99bdf546e7515cd344458cdf6ca525ff4f25865be294bfd5b165ff2fd646d3b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99bdf546e7515cd344458cdf6ca525ff4f25865be294bfd5b165ff2fd646d3b0/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b344834020a2e6cd711f70f0a085ebefc726305e85596a0d4faea0fe1982a6cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b344834020a2e6cd711f70f0a085ebefc726305e85596a0d4faea0fe1982a6cc/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4b11fbdf8c5c21cfa30832d7fa85708793c3b83b5c5590d9a64d94a96fc0695/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4b11fbdf8c5c21cfa30832d7fa85708793c3b83b5c5590d9a64d94a96fc0695/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c82172f79642eea2af2af477954e4b5f6ffb9e0d002a1fb8b2f467ba27b1a474/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c82172f79642eea2af2af477954e4b5f6ffb9e0d002a1fb8b2f467ba27b1a474/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0b99317f8e68051a2ae8abfafcd9bd2d2c108d3150bf46821a327ee5c82fe5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0b99317f8e68051a2ae8abfafcd9bd2d2c108d3150bf46821a327ee5c82fe5e/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e17baf8b373ad3c96a8fc20e38a90c1f6c90b32662a5e44b5db2fb52bf4903d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e17baf8b373ad3c96a8fc20e38a90c1f6c90b32662a5e44b5db2fb52bf4903d0/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2fac9b613796401bcdc48ac22446d0a347cb436c9ab5bfc7201ce36f70cafa3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2fac9b613796401bcdc48ac22446d0a347cb436c9ab5bfc7201ce36f70cafa3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecb4c7cd5ff6e50d991dd72226a5e65034f0b2b66817bb6efdfac434320655fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecb4c7cd5ff6e50d991dd72226a5e65034f0b2b66817bb6efdfac434320655fb/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eddf1c4a5e4f951cd0372023cb686c67eac91db3df8b1d1b1e76ea65c30f82f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eddf1c4a5e4f951cd0372023cb686c67eac91db3df8b1d1b1e76ea65c30f82f7/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef179536fb5e26a2e918dd20de15b387edae7710761ab1622a8b9efc87ff5f36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef179536fb5e26a2e918dd20de15b387edae7710761ab1622a8b9efc87ff5f36/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f692aa15c041ab5ad0a1336f92a892853b45278557eb26993ab47d0378af288f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f692aa15c041ab5ad0a1336f92a892853b45278557eb26993ab47d0378af288f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdc1f1c8a410df54e2792c0e4c8e4a73434731c4d66969e3b87ad47b45a4f88f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdc1f1c8a410df54e2792c0e4c8e4a73434731c4d66969e3b87ad47b45a4f88f/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1294b06a-d192-4f92-bade-e90847b9b3d3/volumes/kubernetes.io~projected/kube-api-access-7q2bm:{mountpoint:/var/lib/kubelet/pods/1294b06a-d192-4f92-bade-e90847b9b3d3/volumes/kubernetes.io~projected/kube-api-access-7q2bm major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1294b06a-d192-4f92-bade-e90847b9b3d3/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1294b06a-d192-4f92-bade-e90847b9b3d3/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/275f0ed9-b587-4bda-996c-56f53df75d68/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/275f0ed9-b587-4bda-996c-56f53df75d68/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/275f0ed9-b587-4bda-996c-56f53df75d68/volumes/kubernetes.io~projected/kube-api-access-b28h6:{mountpoint:/var/lib/kubelet/pods/275f0ed9-b587-4bda-996c-56f53df75d68/volumes/kubernetes.io~projected/kube-api-access-b28h6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/275f0ed9-b587-4bda-996c-56f53df75d68/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/275f0ed9-b587-4bda-996c-56f53df75d68/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes/kubernetes.io~projected/kube-api-access-zzpcr:{mountpoint:/var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes/kubernetes.io~projected/kube-api-access-zzpcr major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volume-subpaths/web-config/config-reloader/2 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volume-subpaths/web-config/prometheus/15 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~empty-dir/config-out major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~projected/kube-api-access-s4qls:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~projected/kube-api-access-s4qls major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~projected/tls-assets major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/config major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/web-config major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/web-config/alertmanager/9 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/web-config/config-reloader/6 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~empty-dir/config-out major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~projected/kube-api-access-5jzh8:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~projected/kube-api-access-5jzh8 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~projected/tls-assets major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/config-volume major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/web-config major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a68ba4-c84a-47ea-812e-0730bec1645e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/62a68ba4-c84a-47ea-812e-0730bec1645e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a68ba4-c84a-47ea-812e-0730bec1645e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/62a68ba4-c84a-47ea-812e-0730bec1645e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a68ba4-c84a-47ea-812e-0730bec1645e/volumes/kubernetes.io~projected/kube-api-access-nj66d:{mountpoint:/var/lib/kubelet/pods/62a68ba4-c84a-47ea-812e-0730bec1645e/volumes/kubernetes.io~projected/kube-api-access-nj66d major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68679646-df21-499f-b5dd-c5b6ceb12314/volumes/kubernetes.io~projected/kube-api-access-kwq8d:{mountpoint:/var/lib/kubelet/pods/68679646-df21-499f-b5dd-c5b6ceb12314/volumes/kubernetes.io~projected/kube-api-access-kwq8d major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68679646-df21-499f-b5dd-c5b6ceb12314/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/68679646-df21-499f-b5dd-c5b6ceb12314/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68679646-df21-499f-b5dd-c5b6ceb12314/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/68679646-df21-499f-b5dd-c5b6ceb12314/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/715adb78-caaa-4665-a368-66bd25143c59/volumes/kubernetes.io~projected/kube-api-access-p6l7h:{mountpoint:/var/lib/kubelet/pods/715adb78-caaa-4665-a368-66bd25143c59/volumes/kubernetes.io~projected/kube-api-access-p6l7h major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/715adb78-caaa-4665-a368-66bd25143c59/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/715adb78-caaa-4665-a368-66bd25143c59/volumes/kubernetes.io~secret/metrics-certs major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80f091be-29ea-430b-a509-e24f1f3169dc/volumes/kubernetes.io~projected/kube-api-access-whnb9:{mountpoint:/var/lib/kubelet/pods/80f091be-29ea-430b-a509-e24f1f3169dc/volumes/kubernetes.io~projected/kube-api-access-whnb9 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d991975-b491-4d47-9345-1c021afcb155/volumes/kubernetes.io~projected/kube-api-access-vn862:{mountpoint:/var/lib/kubelet/pods/8d991975-b491-4d47-9345-1c021afcb155/volumes/kubernetes.io~projected/kube-api-access-vn862 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d991975-b491-4d47-9345-1c021afcb155/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/8d991975-b491-4d47-9345-1c021afcb155/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e01da69-1f87-48ec-9edf-b73ccbf0b244/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8e01da69-1f87-48ec-9edf-b73ccbf0b244/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/944dd579-cc50-401c-a32f-04055024d5f8/volumes/kubernetes.io~projected/kube-api-access-d77fv:{mountpoint:/var/lib/kubelet/pods/944dd579-cc50-401c-a32f-04055024d5f8/volumes/kubernetes.io~projected/kube-api-access-d77fv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/946e5a1e-bfdd-48e2-81a5-cb512473ef85/volumes/kubernetes.io~projected/kube-api-access-prddx:{mountpoint:/var/lib/kubelet/pods/946e5a1e-bfdd-48e2-81a5-cb512473ef85/volumes/kubernetes.io~projected/kube-api-access-prddx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/946e5a1e-bfdd-48e2-81a5-cb512473ef85/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/946e5a1e-bfdd-48e2-81a5-cb512473ef85/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a55d1dbd-a295-4b42-a0fb-b16022237e8c/volumes/kubernetes.io~projected/kube-api-access-2kv9r:{mountpoint:/var/lib/kubelet/pods/a55d1dbd-a295-4b42-a0fb-b16022237e8c/volumes/kubernetes.io~projected/kube-api-access-2kv9r major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5f83ac7-e0a5-47db-bf75-f72a32394313/volumes/kubernetes.io~projected/kube-api-access-xtv7l:{mountpoint:/var/lib/kubelet/pods/b5f83ac7-e0a5-47db-bf75-f72a32394313/volumes/kubernetes.io~projected/kube-api-access-xtv7l major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5f83ac7-e0a5-47db-bf75-f72a32394313/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b5f83ac7-e0a5-47db-bf75-f72a32394313/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5f83ac7-e0a5-47db-bf75-f72a32394313/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b5f83ac7-e0a5-47db-bf75-f72a32394313/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes/kubernetes.io~projected/kube-api-access-9s7k9:{mountpoint:/var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes/kubernetes.io~projected/kube-api-access-9s7k9 major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d54dbca4-a028-4ec4-8aae-6edd83aaff7b/volumes/kubernetes.io~projected/kube-api-access-48h5d:{mountpoint:/var/lib/kubelet/pods/d54dbca4-a028-4ec4-8aae-6edd83aaff7b/volumes/kubernetes.io~projected/kube-api-access-48h5d major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017/volumes/kubernetes.io~projected/kube-api-access-cw8wz:{mountpoint:/var/lib/kubelet/pods/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017/volumes/kubernetes.io~projected/kube-api-access-cw8wz major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edbc19ec-63f1-446d-989e-97fdc12575f7/volumes/kubernetes.io~projected/kube-api-access-rbtvc:{mountpoint:/var/lib/kubelet/pods/edbc19ec-63f1-446d-989e-97fdc12575f7/volumes/kubernetes.io~projected/kube-api-access-rbtvc major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee0ef628-8414-4224-82df-4cede1a6d05b/volumes/kubernetes.io~projected/kube-api-access-jqw98:{mountpoint:/var/lib/kubelet/pods/ee0ef628-8414-4224-82df-4cede1a6d05b/volumes/kubernetes.io~projected/kube-api-access-jqw98 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4015660-eec1-47d8-8432-de3b74803f6a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f4015660-eec1-47d8-8432-de3b74803f6a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:359 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3b2aff40977f9c0391d1c27c946c731144c2f87b656f3d4c5faf7685da46fef1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a9925cdaebf68c12adabe5aa9d35fe949872ff560d2d54b7b02dab668b25f2b2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a9199d779d71cbb8a0d16a21918e70fb8ea1bb53f51388b2759657dd428bb621/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9e1f64ad515ce43bb5b40731208e9698e8a33bd304f0e5e43c691541083ca5dc/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8733710be0352507a3882ef7d366e53e2f0f107a15461435660c75fe7606db6a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c753cbd6db279ac0494f5704e3f5d775eccddebf01cc1f905382bb7c89d21102/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1825740bde5e328ebae5c06b17ec4dc51a8e3fb64904c6cab03f63a40d2a123e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/306ec46a9cf70f5b0682b1efa37b47c49c1a7974f4b341586dff43fe78f6e0b6/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/01cd5fd9c340ab733c20acd732646ca2e113a762f087b9c2684690c0e163a328/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8fd426299e5eb741e493ebc1b150de92feaeffb1451115271e9cba55348a9f26/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/0a8e137a173ab613b69e53eb9caeefdda9b888d57f5f05053aea4cdd4e0f4666/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/300d76f599432cd84d928901be0b2f4e6ade29468f0cb114fdacdee363390990/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/45a8fb269139607ac0e6dcb3ed79120245c599ccfa572b518c39154a04e377f0/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/269b851bf67d1b60431faf051a66b1f70ac9d54f1a25c01c867dfc83e45eb376/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f4cb946db3b424435e8b0d5df35717f1874ba1eb7781c09c6ee319c1a38065bb/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/522bae95674015bec8f7ef9c327ed709fe6869b578ec9e7afb62d601af5deb6c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/259b8b5b31b642b75e298793ea3aaf22cec5bbd1e1a802c820a7d1fa8e0e2344/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0bf6c34620afa751f2eb0a1b2a7ffa54b5881af51981fb99fe077ceae96ea576/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/922be40805aa536d60ebd53bd85b5f2120f282afc3c42785bda81cd78655f7df/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/79078a8f0070c6df4d352c9975be7ec0f2c38124c2f6d6246d900b831fe9fb30/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/06edaf00560df5b7b18ccff903ea6d4b6134d83a514c95dd870b276e368eeb44/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/8e419b55328fc14345e69b65d15406655862bcd5e57a15a42a9dea4c2eb11489/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f0d121055231a43d91fbe6223f30f4a8c087cc4bbfe6e080ee947fb401582fe5/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e0f471784988ba7d2885b01286513fa3c421be52e673409df9b587ead8188457/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/d91cfaabfc16a8d04ebce0d376b2c713d22bdeabd98ff4421b4a94a8799ed07c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/5b88ca95785b118051048e7ca883195ad8ab116b57fe59cc39714b06a631f5cd/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/ff4de176af6a04e2bd6bea82573da768dd24507964c7fc4e80e51edc2436db99/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/1c13765c7ad127e3a380629fdb67b2d1b526f654dec4a05341b42c205ea0d071/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/ca28340a77332b3db16e0ce4b4f62eaf2ee00ad672907e8761e0e7e5199ce500/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/cd286b37144a3088de592cc3ef0d758981c45443f09d2cf8d14e58b57d169dad/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d00f9cdd389644f681d9d73799d18d5ec12924792a6e4f7dd10a56586f790943/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/173bac747ef72475746bfb08ec7a23321699653b9d52e300997901f5b6f38ce1/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e8435123c1a43739c1231423af36fcba3213a3d713d0c40c417e8259d7ed1097/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/291cffe470dcdea854ef67b65c590d449c3f3c118517d1fea9c7a39bf9eaac54/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/a5265eedd4fe8a94b9c4f1789fff93644a8150596e18b3f261a5d9f0d6ffdce8/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/c4bd93e76572623e2379dc75cc25cef10ac2c7ec52b143e09892ca5038f5001b/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/1c6a11286703db0669e2353f98f97c86d6f51617def8f19c58809559e777e1a6/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/210d4f7d21157684498516502937f6759320ce71898968fa2a6455cdcce4aa7b/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/220e2f8980f077c0e7eba8d6208911cd997082163d4ad8e0f4ef99ca5943cb8c/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/1a7a8c7a7293659d186588039445039255e9ead6dcea56ae9d028f0ab48321eb/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/aea5e358d8e9715bf4b1c2904ffa477a48d5f11a11b288363dbf5a3322bbe4ad/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/4a1490383b42b67eb98bdd77afa73541daf3837d044664411baeab2cdb3882c1/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/2ab2e2a12b09c166f700c5cc0f43118a086da4d00eaba7228797694c967f8595/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/5bd9397ce43e6c6201647f9f9953885e1d1fb7e6992c9ef407c3385557396155/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/11dca8e0605d0ea9022e79c7f5c2d4315b5ddfcb5bd02cd5e538c7ec43d2c2f1/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/c8fe1c989aa046a4458add5c2ba9c32ff772511e6a6934550b70fcc3a64ec9c2/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/384f334c089468da885a766217b66d9aa8b03149b4d9ad60cf8656a317660286/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/5ce857c2ad2565c670e00bbf2abeeddc13b05780eb75ceee6bfa813d3069400d/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/a69c1f4ce15ff22be64c2d1d452f795f1e956bb7fab4be668544176eb55261e0/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/0b6680e99239c61a398ecaf59b2ecb06fb990a305e039f4a5b839b612e4e86a6/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/e863c3140d5b7720d35b12ce6227ece02aa7ca826262256072cb39cab654f56f/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/b04c7c6fa26d64442ec54c04e0ce77728059d2f97468c06dece8c8c124e35666/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/d7795694071fb2645bbc2394b742ed5f0a1ad164adedbacc44c91140eb5dd24e/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/d4c1d8c527473ef1b392d1cc261ce6ba6fb0e1cfdd4c6943f1fc376d2bc8d527/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/20df47e94962c887aefbb22871bb765d03fcb90998efcacb0773c145cef17ca6/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/2b7cb43a8990a7443e3f66591545c09d186c23b4c6e02c2c62a5bf1a8ce7f178/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/78219561b9385b7d25634fdaa27b2874a8632e4ce1b308b9f359b1815713a88e/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/f8039eab7794aedb8ef42c3caca2820eed9c5c07c8ee4f557bd0e3fafe3434e3/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/71d88d02bd1b64e816d4d5cccbb3e18dca6d323eebb51165db92c532379036cc/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/15d0d79a6fba27b71a4014c3a2d8a01425f69a85d8f9e13be59c5208daff117c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/86a14647896849220c113f8cb5744234d65bdf2c6aa55eb1e70953a0b4a0a115/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/45557b19d361bc2f758b84bfab79af42e9c2fb775cea34c2854248ae47701649/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/60d109277cba10791a02344066413136679f0ed62297d877c8a1062e1ef1a5e9/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/362aebab7df06fbe8d654d7a7a7a86ab19870d64464af7ce2a3ae43426e2b775/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/0e7882539039082fcb56b9676adfbd2bb8407f2021962a876c4dff02099c4f31/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/9c332b2ccbb7906dce70c7234bc242e0377c3f864f74f314bd8815bac9fd2e0b/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/2f8e74812bb1540c3becfd95836dee9194d57e7aa1c5af4143f7307d654d3b78/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 02 08:06:41.519828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.519385 6160 manager.go:217] Machine: {Timestamp:2026-11-02 08:06:41.518806003 +0000 UTC m=+0.072310193 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100572 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cb327eb33dfebd9749aee3faeaf2b SystemUUID:ec2cb327-eb33-dfeb-d974-9aee3faeaf2b BootID:9bfb9162-f2a2-4ee8-b8f0-659d50b5a507 Filesystems:[{Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:292 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d54dbca4-a028-4ec4-8aae-6edd83aaff7b/volumes/kubernetes.io~projected/kube-api-access-48h5d DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d991975-b491-4d47-9345-1c021afcb155/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/68679646-df21-499f-b5dd-c5b6ceb12314/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:251 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b344834020a2e6cd711f70f0a085ebefc726305e85596a0d4faea0fe1982a6cc/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62a68ba4-c84a-47ea-812e-0730bec1645e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes/kubernetes.io~projected/kube-api-access-zzpcr DeviceMajor:0 DeviceMinor:376 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdc1f1c8a410df54e2792c0e4c8e4a73434731c4d66969e3b87ad47b45a4f88f/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~projected/kube-api-access-s4qls DeviceMajor:0 DeviceMinor:349 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee0ef628-8414-4224-82df-4cede1a6d05b/volumes/kubernetes.io~projected/kube-api-access-jqw98 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecb4c7cd5ff6e50d991dd72226a5e65034f0b2b66817bb6efdfac434320655fb/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:374 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82f719c36611a6097abc16cf244a1295a4b693ef3dbd19d6211583308d4c0757/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82d1d3c0042721b0af053546d841ca6dafea3c07347eba6e08353ad36cde2b8b/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e17baf8b373ad3c96a8fc20e38a90c1f6c90b32662a5e44b5db2fb52bf4903d0/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/674d2b682e37c20c3919821046c794399acd8984e5484b759f1ed0a9b15e1eeb/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/275f0ed9-b587-4bda-996c-56f53df75d68/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e01da69-1f87-48ec-9edf-b73ccbf0b244/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/944dd579-cc50-401c-a32f-04055024d5f8/volumes/kubernetes.io~projected/kube-api-access-d77fv DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5f83ac7-e0a5-47db-bf75-f72a32394313/volumes/kubernetes.io~projected/kube-api-access-xtv7l DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edbc19ec-63f1-446d-989e-97fdc12575f7/volumes/kubernetes.io~projected/kube-api-access-rbtvc DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:344 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a55d1dbd-a295-4b42-a0fb-b16022237e8c/volumes/kubernetes.io~projected/kube-api-access-2kv9r DeviceMajor:0 DeviceMinor:361 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fcf0e7c7a1196d3df280b0c683f5b57d2f71dacd035cd2fe08bc1c5c78ee619/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8d991975-b491-4d47-9345-1c021afcb155/volumes/kubernetes.io~projected/kube-api-access-vn862 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5f83ac7-e0a5-47db-bf75-f72a32394313/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70f2921a186f179e92aa7d509524b1302dbf6d0b7e3f26cba55a9e2b9644c307/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4b11fbdf8c5c21cfa30832d7fa85708793c3b83b5c5590d9a64d94a96fc0695/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef179536fb5e26a2e918dd20de15b387edae7710761ab1622a8b9efc87ff5f36/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:290 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c82172f79642eea2af2af477954e4b5f6ffb9e0d002a1fb8b2f467ba27b1a474/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/946e5a1e-bfdd-48e2-81a5-cb512473ef85/volumes/kubernetes.io~projected/kube-api-access-prddx DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017/volumes/kubernetes.io~projected/kube-api-access-cw8wz DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45f95867ea5f3f86697e78995bc373467e88e894eb245f262be2e65deaf2c9a3/userdata/shm DeviceMajor:0 DeviceMinor:268 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:296 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~projected/kube-api-access-5jzh8 DeviceMajor:0 DeviceMinor:297 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/275f0ed9-b587-4bda-996c-56f53df75d68/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1294b06a-d192-4f92-bade-e90847b9b3d3/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/715adb78-caaa-4665-a368-66bd25143c59/volumes/kubernetes.io~projected/kube-api-access-p6l7h DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5f83ac7-e0a5-47db-bf75-f72a32394313/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:341 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62a68ba4-c84a-47ea-812e-0730bec1645e/volumes/kubernetes.io~projected/kube-api-access-nj66d DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70e067f8cdfb3cadac3af0a9ba448a7beb9a0fc9dba1c72274d6fdd9f45d23b3/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/715adb78-caaa-4665-a368-66bd25143c59/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:360 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68679646-df21-499f-b5dd-c5b6ceb12314/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/393a34c14ee09719bfdab0517a1b5cac495b200fd4fa3f6d8958e7e3ca608cad/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f692aa15c041ab5ad0a1336f92a892853b45278557eb26993ab47d0378af288f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68679646-df21-499f-b5dd-c5b6ceb12314/volumes/kubernetes.io~projected/kube-api-access-kwq8d DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes/kubernetes.io~projected/kube-api-access-9s7k9 DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4015660-eec1-47d8-8432-de3b74803f6a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:359 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99bdf546e7515cd344458cdf6ca525ff4f25865be294bfd5b165ff2fd646d3b0/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/646adad95a53d1ac07326d5ca32a8c5f169d7220beab88980809d0bd77845583/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80f091be-29ea-430b-a509-e24f1f3169dc/volumes/kubernetes.io~projected/kube-api-access-whnb9 DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0b99317f8e68051a2ae8abfafcd9bd2d2c108d3150bf46821a327ee5c82fe5e/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:290 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:375 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62a68ba4-c84a-47ea-812e-0730bec1645e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/275f0ed9-b587-4bda-996c-56f53df75d68/volumes/kubernetes.io~projected/kube-api-access-b28h6 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2fac9b613796401bcdc48ac22446d0a347cb436c9ab5bfc7201ce36f70cafa3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/946e5a1e-bfdd-48e2-81a5-cb512473ef85/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eddf1c4a5e4f951cd0372023cb686c67eac91db3df8b1d1b1e76ea65c30f82f7/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8635ab5968e926289e2ff01e43c2988edbc62ef85663db5b03d7721ca42b8b5a/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:250 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1294b06a-d192-4f92-bade-e90847b9b3d3/volumes/kubernetes.io~projected/kube-api-access-7q2bm DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2fcf0e7c7a1196d MacAddress:d2:29:78:a8:49:a3 Speed:10000 Mtu:8901} {Name:393a34c14ee0971 MacAddress:c2:35:47:9c:22:18 Speed:10000 Mtu:8901} {Name:45f95867ea5f3f8 MacAddress:06:8a:a3:5d:e9:99 Speed:10000 Mtu:8901} {Name:674d2b682e37c20 MacAddress:ee:6d:76:6f:06:75 Speed:10000 Mtu:8901} {Name:70e067f8cdfb3ca MacAddress:fe:bc:09:e7:b3:d0 Speed:10000 Mtu:8901} {Name:70f2921a186f179 MacAddress:26:bc:5b:35:74:8c Speed:10000 Mtu:8901} {Name:82d1d3c0042721b MacAddress:ea:37:6b:a7:65:76 Speed:10000 Mtu:8901} {Name:8635ab5968e9262 MacAddress:36:d9:9e:c3:cb:41 Speed:10000 Mtu:8901} {Name:99bdf546e7515cd MacAddress:22:76:25:31:5f:2b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e8:16:16:fb Speed:0 Mtu:9001} {Name:br-int MacAddress:86:39:b5:4b:06:37 Speed:0 Mtu:8901} {Name:d0b99317f8e6805 MacAddress:26:b6:4f:c1:5c:a3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e8:16:16:fb Speed:0 Mtu:9001} {Name:f692aa15c041ab5 MacAddress:96:f8:96:cb:9c:4e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fa:f8:e3:81:ed:69 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:41:a5:ce:42:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 08:06:41.520101 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.519834 6160 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 08:06:41.520101 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.519887 6160 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 08:06:41.520318 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.520292 6160 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 08:06:41.520474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.520319 6160 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-103.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 08:06:41.520552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.520479 6160 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 08:06:41.520552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.520492 6160 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 08:06:41.520552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.520506 6160 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 08:06:41.520552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.520534 6160 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 08:06:41.520705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.520677 6160 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 08:06:41.520923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.520908 6160 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 08:06:41.521043 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.521031 6160 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 08:06:41.521096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.521079 6160 kubelet.go:493] "Attempting to sync node with API server" Nov 02 08:06:41.521096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.521094 6160 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 08:06:41.521208 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.521108 6160 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 08:06:41.521208 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.521124 6160 kubelet.go:405] "Adding apiserver pod source" Nov 02 08:06:41.521208 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.521145 6160 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 08:06:41.521959 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.521942 6160 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 08:06:41.521959 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.521959 6160 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 08:06:41.522543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.522522 6160 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 08:06:41.535164 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535140 6160 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 08:06:41.535164 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535161 6160 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 08:06:41.535307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535291 6160 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 08:06:41.535533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535519 6160 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 08:06:41.535585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535557 6160 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 08:06:41.535678 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535667 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 08:06:41.535727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535683 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 08:06:41.535727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535690 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 08:06:41.535727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535700 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 08:06:41.535727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535710 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 08:06:41.535727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535720 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 08:06:41.535727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535727 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 08:06:41.535928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535733 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 08:06:41.535928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535740 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 08:06:41.535928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535745 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 08:06:41.535928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535758 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 08:06:41.535928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535779 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 08:06:41.535928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535814 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 08:06:41.535928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.535823 6160 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 08:06:41.539974 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.539958 6160 server.go:1263] "Started kubelet" Nov 02 08:06:41.541230 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.540021 6160 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 08:06:41.541230 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.540160 6160 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 08:06:41.541230 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.540281 6160 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 08:06:41.541230 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.540771 6160 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 08:06:41.540518 ip-10-0-136-103 systemd[1]: Started Kubernetes Kubelet. Nov 02 08:06:41.545526 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.545169 6160 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 08:06:41.545629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.545615 6160 server.go:317] "Adding debug handlers to kubelet server" Nov 02 08:06:41.545760 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:41.545723 6160 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-103.us-west-2.compute.internal\" not found" Nov 02 08:06:41.545760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.545725 6160 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 08:06:41.545892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.545808 6160 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 08:06:41.545892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.545817 6160 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 08:06:41.546086 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.546049 6160 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 08:00:29 +0000 UTC" deadline="2028-04-04 20:35:59.617628963 +0000 UTC" Nov 02 08:06:41.546163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.546088 6160 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12468h29m18.071543868s" Nov 02 08:06:41.546163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.546140 6160 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 08:06:41.549312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549248 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1294b06a-d192-4f92-bade-e90847b9b3d3" volumeName="kubernetes.io/empty-dir/1294b06a-d192-4f92-bade-e90847b9b3d3-tmp-dir" seLinuxMountContext="" Nov 02 08:06:41.549312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549283 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 08:06:41.549312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549300 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 08:06:41.549312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549313 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5f83ac7-e0a5-47db-bf75-f72a32394313" volumeName="kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-tls" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549325 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" volumeName="kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549338 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="275f0ed9-b587-4bda-996c-56f53df75d68" volumeName="kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-config" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549352 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="275f0ed9-b587-4bda-996c-56f53df75d68" volumeName="kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-env-overrides" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549364 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="275f0ed9-b587-4bda-996c-56f53df75d68" volumeName="kubernetes.io/secret/275f0ed9-b587-4bda-996c-56f53df75d68-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549378 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" volumeName="kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549386 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549394 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944dd579-cc50-401c-a32f-04055024d5f8" volumeName="kubernetes.io/configmap/944dd579-cc50-401c-a32f-04055024d5f8-serviceca" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549402 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="946e5a1e-bfdd-48e2-81a5-cb512473ef85" volumeName="kubernetes.io/secret/946e5a1e-bfdd-48e2-81a5-cb512473ef85-cert" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549410 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" volumeName="kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549429 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549440 6160 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549446 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a68ba4-c84a-47ea-812e-0730bec1645e" volumeName="kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-tuned" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549452 6160 factory.go:55] Registering systemd factory Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549458 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d991975-b491-4d47-9345-1c021afcb155" volumeName="kubernetes.io/configmap/8d991975-b491-4d47-9345-1c021afcb155-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549473 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944dd579-cc50-401c-a32f-04055024d5f8" volumeName="kubernetes.io/projected/944dd579-cc50-401c-a32f-04055024d5f8-kube-api-access-d77fv" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549486 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4015660-eec1-47d8-8432-de3b74803f6a" volumeName="kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549499 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" volumeName="kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549510 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549523 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549533 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549544 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8" seLinuxMountContext="" Nov 02 08:06:41.549574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549574 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68679646-df21-499f-b5dd-c5b6ceb12314" volumeName="kubernetes.io/configmap/68679646-df21-499f-b5dd-c5b6ceb12314-metrics-client-ca" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549591 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="715adb78-caaa-4665-a368-66bd25143c59" volumeName="kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549606 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80f091be-29ea-430b-a509-e24f1f3169dc" volumeName="kubernetes.io/projected/80f091be-29ea-430b-a509-e24f1f3169dc-kube-api-access-whnb9" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549463 6160 factory.go:223] Registration of the systemd container factory successfully Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549623 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="275f0ed9-b587-4bda-996c-56f53df75d68" volumeName="kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-script-lib" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549635 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549675 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549690 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549705 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549718 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549731 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549744 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549755 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549767 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="715adb78-caaa-4665-a368-66bd25143c59" volumeName="kubernetes.io/projected/715adb78-caaa-4665-a368-66bd25143c59-kube-api-access-p6l7h" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549777 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d991975-b491-4d47-9345-1c021afcb155" volumeName="kubernetes.io/projected/8d991975-b491-4d47-9345-1c021afcb155-kube-api-access-vn862" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549789 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a55d1dbd-a295-4b42-a0fb-b16022237e8c" volumeName="kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549802 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5f83ac7-e0a5-47db-bf75-f72a32394313" volumeName="kubernetes.io/empty-dir/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-textfile" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549811 6160 factory.go:153] Registering CRI-O factory Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549829 6160 factory.go:223] Registration of the crio container factory successfully Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549848 6160 factory.go:103] Registering Raw factory Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549863 6160 manager.go:1199] Started watching for new ooms in manager Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550202 6160 manager.go:319] Starting recovery of all containers Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.549814 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5f83ac7-e0a5-47db-bf75-f72a32394313" volumeName="kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550495 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1294b06a-d192-4f92-bade-e90847b9b3d3" volumeName="kubernetes.io/projected/1294b06a-d192-4f92-bade-e90847b9b3d3-kube-api-access-7q2bm" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550520 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="275f0ed9-b587-4bda-996c-56f53df75d68" volumeName="kubernetes.io/projected/275f0ed9-b587-4bda-996c-56f53df75d68-kube-api-access-b28h6" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550536 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" volumeName="kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550555 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550568 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550587 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550603 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d991975-b491-4d47-9345-1c021afcb155" volumeName="kubernetes.io/empty-dir/8d991975-b491-4d47-9345-1c021afcb155-data-volume" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550616 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71abbe8-6a3a-4d0f-9a55-6fe051cf0017" volumeName="kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cni-binary-copy" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550628 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1294b06a-d192-4f92-bade-e90847b9b3d3" volumeName="kubernetes.io/configmap/1294b06a-d192-4f92-bade-e90847b9b3d3-config-volume" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550639 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550650 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 08:06:41.550644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550662 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550675 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550687 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="946e5a1e-bfdd-48e2-81a5-cb512473ef85" volumeName="kubernetes.io/projected/946e5a1e-bfdd-48e2-81a5-cb512473ef85-kube-api-access-prddx" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550699 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5f83ac7-e0a5-47db-bf75-f72a32394313" volumeName="kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550711 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" volumeName="kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550734 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" volumeName="kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550764 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550777 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550788 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80f091be-29ea-430b-a509-e24f1f3169dc" volumeName="kubernetes.io/configmap/80f091be-29ea-430b-a509-e24f1f3169dc-iptables-alerter-script" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550799 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d991975-b491-4d47-9345-1c021afcb155" volumeName="kubernetes.io/secret/8d991975-b491-4d47-9345-1c021afcb155-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550809 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" volumeName="kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550821 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d54dbca4-a028-4ec4-8aae-6edd83aaff7b" volumeName="kubernetes.io/empty-dir/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-tmp-dir" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550832 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d54dbca4-a028-4ec4-8aae-6edd83aaff7b" volumeName="kubernetes.io/projected/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-kube-api-access-48h5d" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550846 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" volumeName="kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550858 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550870 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550881 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edbc19ec-63f1-446d-989e-97fdc12575f7" volumeName="kubernetes.io/projected/edbc19ec-63f1-446d-989e-97fdc12575f7-kube-api-access-rbtvc" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550894 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0ef628-8414-4224-82df-4cede1a6d05b" volumeName="kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-binary-copy" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550905 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0ef628-8414-4224-82df-4cede1a6d05b" volumeName="kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550918 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0ef628-8414-4224-82df-4cede1a6d05b" volumeName="kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550933 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee0ef628-8414-4224-82df-4cede1a6d05b" volumeName="kubernetes.io/projected/ee0ef628-8414-4224-82df-4cede1a6d05b-kube-api-access-jqw98" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550945 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550957 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a68ba4-c84a-47ea-812e-0730bec1645e" volumeName="kubernetes.io/projected/62a68ba4-c84a-47ea-812e-0730bec1645e-kube-api-access-nj66d" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550968 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68679646-df21-499f-b5dd-c5b6ceb12314" volumeName="kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-tls" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550981 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.550994 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68679646-df21-499f-b5dd-c5b6ceb12314" volumeName="kubernetes.io/projected/68679646-df21-499f-b5dd-c5b6ceb12314-kube-api-access-kwq8d" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551004 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" volumeName="kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551027 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a68ba4-c84a-47ea-812e-0730bec1645e" volumeName="kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-tmp" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551047 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68679646-df21-499f-b5dd-c5b6ceb12314" volumeName="kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551070 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5f83ac7-e0a5-47db-bf75-f72a32394313" volumeName="kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-metrics-client-ca" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551089 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" volumeName="kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551108 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" volumeName="kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551125 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" volumeName="kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551140 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551157 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551201 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e01da69-1f87-48ec-9edf-b73ccbf0b244" volumeName="kubernetes.io/configmap/8e01da69-1f87-48ec-9edf-b73ccbf0b244-konnectivity-ca" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551213 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e01da69-1f87-48ec-9edf-b73ccbf0b244" volumeName="kubernetes.io/secret/8e01da69-1f87-48ec-9edf-b73ccbf0b244-agent-certs" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551226 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" volumeName="kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551237 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71abbe8-6a3a-4d0f-9a55-6fe051cf0017" volumeName="kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-daemon-config" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551248 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1294b06a-d192-4f92-bade-e90847b9b3d3" volumeName="kubernetes.io/secret/1294b06a-d192-4f92-bade-e90847b9b3d3-metrics-tls" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551259 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" volumeName="kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551270 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40a4044b-18d1-494c-b50e-2b3d253efdf2" volumeName="kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551283 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5f83ac7-e0a5-47db-bf75-f72a32394313" volumeName="kubernetes.io/projected/b5f83ac7-e0a5-47db-bf75-f72a32394313-kube-api-access-xtv7l" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551295 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71abbe8-6a3a-4d0f-9a55-6fe051cf0017" volumeName="kubernetes.io/projected/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-kube-api-access-cw8wz" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551307 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a387bc4-fe09-4921-9ff4-8199642acb38" volumeName="kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551318 6160 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" volumeName="kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca" seLinuxMountContext="" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551330 6160 reconstruct.go:97] "Volume reconstruction finished" Nov 02 08:06:41.552197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.551338 6160 reconciler.go:29] "Reconciler: start to sync state" Nov 02 08:06:41.573559 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.573498 6160 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 08:06:41.575495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.575468 6160 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 08:06:41.575979 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.575959 6160 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 08:06:41.576075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.575993 6160 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 08:06:41.576075 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:41.576039 6160 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 08:06:41.588050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.588004 6160 generic.go:356] "Generic (PLEG): container finished" podID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" exitCode=0 Nov 02 08:06:41.596406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.596374 6160 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="2cefb1187ca75da12f8d7420bda38bd57b547027d58f4f708ce234fd58f2f10c" exitCode=0 Nov 02 08:06:41.596528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.596412 6160 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="008db88bdd69666a8bb8848d9fadc04f1f80e9523f89ea4398c69767d48ee44b" exitCode=0 Nov 02 08:06:41.596528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.596426 6160 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="c94db155f2d5fbca4dac26294d0b1cab559619958658740023b95d048d0ce808" exitCode=0 Nov 02 08:06:41.596528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.596434 6160 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="5d14d86cdc8092d6d5cd7d4ad1ee4a9d92a5c832ba2d313aaf3afb5aed9d5de3" exitCode=0 Nov 02 08:06:41.596528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.596441 6160 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="083ef05e2a4139e16d8830ab6d2fbf326e1137ca16cc5ffe9e4332ff85444783" exitCode=0 Nov 02 08:06:41.596528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.596449 6160 generic.go:356] "Generic (PLEG): container finished" podID="ee0ef628-8414-4224-82df-4cede1a6d05b" containerID="9bb62d368d9b42d9fa9f625eb72eab80c4866b292d472566bb9c062fe9e6bf60" exitCode=0 Nov 02 08:06:41.630631 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.630553 6160 generic.go:356] "Generic (PLEG): container finished" podID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerID="a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be" exitCode=0 Nov 02 08:06:41.634930 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.634894 6160 generic.go:356] "Generic (PLEG): container finished" podID="b5f83ac7-e0a5-47db-bf75-f72a32394313" containerID="2906b926913e5094f13178c10646167e7534aca2ec6eae80f6b5923eea770e5b" exitCode=0 Nov 02 08:06:41.645601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.645567 6160 generic.go:356] "Generic (PLEG): container finished" podID="0551724789a3970575b47d5ece2444c4" containerID="fd739ec8c0a5113501b1bc521c0a10042a5a83f72afd897bd2eb4855f20e1f5c" exitCode=0 Nov 02 08:06:41.645792 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:41.645773 6160 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-103.us-west-2.compute.internal\" not found" Nov 02 08:06:41.668212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.668161 6160 manager.go:324] Recovery completed Nov 02 08:06:41.672621 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.672598 6160 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:06:41.676223 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:41.676195 6160 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 08:06:41.695660 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.695635 6160 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:06:41.697854 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.697833 6160 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:06:41.697976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.697868 6160 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:06:41.697976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.697884 6160 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:06:41.702847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.702826 6160 cpu_manager.go:226] "Starting" policy="none" Nov 02 08:06:41.702847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.702846 6160 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 08:06:41.703009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.702866 6160 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 08:06:41.703074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703055 6160 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 08:06:41.703119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703076 6160 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 08:06:41.703119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703100 6160 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 08:06:41.703119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703109 6160 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 08:06:41.703119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703116 6160 policy_none.go:50] "Start" Nov 02 08:06:41.703303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703152 6160 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 08:06:41.703303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703165 6160 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 08:06:41.703420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703405 6160 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 08:06:41.703476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703422 6160 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 08:06:41.703476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.703430 6160 policy_none.go:44] "Start" Nov 02 08:06:41.704632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.704612 6160 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:06:41.706150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.706131 6160 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:06:41.706273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.706211 6160 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:06:41.706273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.706229 6160 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:06:41.710397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.710369 6160 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 08:06:41.714217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.714198 6160 manager.go:341] "Starting Device Plugin manager" Nov 02 08:06:41.714301 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:41.714233 6160 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 08:06:41.714301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.714247 6160 server.go:87] "Starting device plugin registration server" Nov 02 08:06:41.714572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.714556 6160 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 08:06:41.715167 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.714574 6160 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 08:06:41.715382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.715363 6160 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 08:06:41.717238 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:41.717215 6160 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 08:06:41.717331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.717228 6160 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 08:06:41.717331 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:41.717257 6160 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-103.us-west-2.compute.internal\" not found" Nov 02 08:06:41.717439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.717408 6160 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 08:06:41.717439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.717428 6160 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 08:06:41.717535 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.717484 6160 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:06:41.717572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.717480 6160 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T08:06:41.717262706Z","UUID":"c8abb604-0b95-4710-968a-b778d8fb1185","Handler":null,"Name":"","Endpoint":""} Nov 02 08:06:41.720828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.720808 6160 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 08:06:41.720915 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.720847 6160 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 08:06:41.763146 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.763116 6160 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:06:41.811568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.811537 6160 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 08:06:41.811774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.811640 6160 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 08:06:41.815883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.815857 6160 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 08:06:41.816995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.816968 6160 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 08:06:41.817093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.817003 6160 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 08:06:41.817093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.817015 6160 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-103.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 08:06:41.817168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.817093 6160 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:41.876682 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.876621 6160 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal"] Nov 02 08:06:41.877467 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.877448 6160 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:41.877559 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.877519 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"fdbcb27b80ac4a4b19ed5f7d7dc1c25f","Type":"ContainerStarted","Data":"feaa6c4e9d9dfd877f01c4371e42994b0c5daf2d9b788555500bdaf7fe88927c"} Nov 02 08:06:41.877594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.877577 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"fdbcb27b80ac4a4b19ed5f7d7dc1c25f","Type":"ContainerStarted","Data":"ef179536fb5e26a2e918dd20de15b387edae7710761ab1622a8b9efc87ff5f36"} Nov 02 08:06:41.877659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.877636 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"0551724789a3970575b47d5ece2444c4","Type":"ContainerStarted","Data":"81fa01985a6af788a9e596f7e7618a2abefb15d88be7601f969ec93f58d60b47"} Nov 02 08:06:41.877700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.877663 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"0551724789a3970575b47d5ece2444c4","Type":"ContainerDied","Data":"fd739ec8c0a5113501b1bc521c0a10042a5a83f72afd897bd2eb4855f20e1f5c"} Nov 02 08:06:41.877700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.877693 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" event={"ID":"0551724789a3970575b47d5ece2444c4","Type":"ContainerStarted","Data":"646adad95a53d1ac07326d5ca32a8c5f169d7220beab88980809d0bd77845583"} Nov 02 08:06:41.877790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.877759 6160 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:41.917578 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.917537 6160 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:41.917744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.917626 6160 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:41.935552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.935508 6160 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:06:41.935716 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:41.935576 6160 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:41.936093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:41.935982 6160 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:06:41.936093 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:41.936035 6160 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.012308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.012275 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdbcb27b80ac4a4b19ed5f7d7dc1c25f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"fdbcb27b80ac4a4b19ed5f7d7dc1c25f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.012463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.012317 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.012463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.012350 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.113264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.113225 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdbcb27b80ac4a4b19ed5f7d7dc1c25f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"fdbcb27b80ac4a4b19ed5f7d7dc1c25f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.113407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.113283 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.113407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.113303 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.113407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.113245 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdbcb27b80ac4a4b19ed5f7d7dc1c25f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"fdbcb27b80ac4a4b19ed5f7d7dc1c25f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.113407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.113347 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.113407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.113380 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0551724789a3970575b47d5ece2444c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" (UID: \"0551724789a3970575b47d5ece2444c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.522005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.521959 6160 apiserver.go:52] "Watching apiserver" Nov 02 08:06:42.654925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.654896 6160 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.655069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.655037 6160 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.665185 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.665158 6160 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 08:06:42.665842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.665821 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-82pzl","openshift-monitoring/alertmanager-main-0","kube-system/konnectivity-agent-7jfqx","openshift-dns/node-resolver-ndqtq","openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6","openshift-insights/insights-runtime-extractor-kck72","openshift-monitoring/prometheus-k8s-0","openshift-network-operator/iptables-alerter-5g226","kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm","openshift-console/console-6bcd5f847b-zrw55","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-rshcn","openshift-cluster-node-tuning-operator/tuned-ppnkn","openshift-ovn-kubernetes/ovnkube-node-2r44q","openshift-ingress-canary/ingress-canary-jddzv","openshift-monitoring/node-exporter-m89dq","openshift-multus/network-metrics-daemon-zjzfb","openshift-dns/dns-default-l7nzr","openshift-multus/multus-rclzx","openshift-console/console-5dd89b74f-67l4d","kube-system/global-pull-secret-syncer-xs697","openshift-image-registry/node-ca-8v9wz"] Nov 02 08:06:42.729125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.729096 6160 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:06:42.729307 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:42.729192 6160 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.730037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.730007 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.730190 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.730143 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 08:06:42.730298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.730231 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 08:06:42.730298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.730241 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.730298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.730275 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.730298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.730279 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.730490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.730277 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.730490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.730303 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 08:06:42.730490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.730349 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 08:06:42.731240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.731218 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.731416 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.731229 6160 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 08:06:42.731504 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:06:42.731480 6160 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-103.us-west-2.compute.internal" Nov 02 08:06:42.731782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.731674 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.731782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.731721 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 08:06:42.731918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.731802 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.731918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.731860 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-shlrq\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732027 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732109 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732122 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732264 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rwld\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732367 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732473 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732559 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghx8m\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732579 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zrnf9\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732619 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2bt7\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732650 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732732 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qmmzq\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732730 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nct\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732563 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732799 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.732944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.732884 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bgzdd\"" Nov 02 08:06:42.733642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.733007 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-c87zx\"" Nov 02 08:06:42.733642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.733050 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 08:06:42.733642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.733083 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwch9\"" Nov 02 08:06:42.748186 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.748161 6160 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 08:06:42.755401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.755383 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 08:06:42.755469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.755454 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 08:06:42.755668 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.755654 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 08:06:42.755728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.755682 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 08:06:42.755728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.755722 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 08:06:42.755803 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.755737 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 08:06:42.755803 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.755754 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4v7vr\"" Nov 02 08:06:42.755803 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.755775 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t9rxp\"" Nov 02 08:06:42.760283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.760269 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.760337 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.760283 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 08:06:42.760429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.760418 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.776058 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.776003 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 08:06:42.776058 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.776051 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 08:06:42.776163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.776077 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.776312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.776301 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 08:06:42.777620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.777607 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 08:06:42.777804 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.777793 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 08:06:42.801845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.801823 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 08:06:42.801845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.801840 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 08:06:42.801936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.801851 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 08:06:42.802494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.802478 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.802573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.802530 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 08:06:42.806950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.806843 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 08:06:42.806950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.806878 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 08:06:42.806950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.806929 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.807109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.807064 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zgn8x\"" Nov 02 08:06:42.807139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.807111 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lbg2j\"" Nov 02 08:06:42.817442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817420 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.817517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817449 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.817517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817468 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.817596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817556 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.817650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817592 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpcr\" (UniqueName: \"kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.817650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817633 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwq8d\" (UniqueName: \"kubernetes.io/projected/68679646-df21-499f-b5dd-c5b6ceb12314-kube-api-access-kwq8d\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:42.817747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817668 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80f091be-29ea-430b-a509-e24f1f3169dc-host-slash\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:06:42.817747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817689 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.817747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817697 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-sys\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.817747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817725 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.817908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817836 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.817908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817878 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6l7h\" (UniqueName: \"kubernetes.io/projected/715adb78-caaa-4665-a368-66bd25143c59-kube-api-access-p6l7h\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:42.818000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817906 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48h5d\" (UniqueName: \"kubernetes.io/projected/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-kube-api-access-48h5d\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:06:42.818000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817935 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-log-socket\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.818000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.817953 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-dbus\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:42.818141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818067 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.818141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818105 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-bin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.818141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818130 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.818314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818161 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.818314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818232 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4qls\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.818314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818299 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.818468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818323 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e01da69-1f87-48ec-9edf-b73ccbf0b244-agent-certs\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:06:42.818468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818329 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.818468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818340 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e01da69-1f87-48ec-9edf-b73ccbf0b244-konnectivity-ca\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:06:42.818468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818363 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-socket-dir-parent\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818481 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-device-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818500 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-tuned\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818516 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-node-log\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818516 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818537 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-env-overrides\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818566 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-textfile\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818585 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e01da69-1f87-48ec-9edf-b73ccbf0b244-agent-certs\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818603 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e01da69-1f87-48ec-9edf-b73ccbf0b244-konnectivity-ca\") pod \"konnectivity-agent-7jfqx\" (UID: \"8e01da69-1f87-48ec-9edf-b73ccbf0b244\") " pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818613 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-netns\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818623 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818632 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-tuned\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.818659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818649 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28h6\" (UniqueName: \"kubernetes.io/projected/275f0ed9-b587-4bda-996c-56f53df75d68-kube-api-access-b28h6\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818675 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-textfile\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818682 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818782 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzh8\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818800 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818825 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cni-binary-copy\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818851 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-k8s-cni-cncf-io\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818872 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-ovn\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818910 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818943 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.818954 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cni-binary-copy\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819036 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-sys-fs\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.819083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819065 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77fv\" (UniqueName: \"kubernetes.io/projected/944dd579-cc50-401c-a32f-04055024d5f8-kube-api-access-d77fv\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819101 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-slash\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819143 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819217 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819260 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d991975-b491-4d47-9345-1c021afcb155-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819289 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1294b06a-d192-4f92-bade-e90847b9b3d3-metrics-tls\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819360 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-modprobe-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819386 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-systemd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819403 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-metrics-client-ca\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819426 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqw98\" (UniqueName: \"kubernetes.io/projected/ee0ef628-8414-4224-82df-4cede1a6d05b-kube-api-access-jqw98\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819468 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-systemd\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819597 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-var-lib-kubelet\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.819617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819618 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946e5a1e-bfdd-48e2-81a5-cb512473ef85-cert\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819644 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819672 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-metrics-client-ca\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819708 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819768 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819784 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819798 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtv7l\" (UniqueName: \"kubernetes.io/projected/b5f83ac7-e0a5-47db-bf75-f72a32394313-kube-api-access-xtv7l\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819813 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946e5a1e-bfdd-48e2-81a5-cb512473ef85-cert\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819819 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819842 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944dd579-cc50-401c-a32f-04055024d5f8-host\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819920 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-tmp-dir\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819941 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819955 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.819977 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-etc-kubernetes\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820000 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820003 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbtvc\" (UniqueName: \"kubernetes.io/projected/edbc19ec-63f1-446d-989e-97fdc12575f7-kube-api-access-rbtvc\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820022 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-tmp-dir\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820052 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.820078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820067 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944dd579-cc50-401c-a32f-04055024d5f8-serviceca\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820094 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820138 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-wtmp\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820166 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820224 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820242 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820279 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-conf-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820333 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-tmp\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820359 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820374 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-hosts-file\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820402 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820414 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a68ba4-c84a-47ea-812e-0730bec1645e-tmp\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820440 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820467 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1294b06a-d192-4f92-bade-e90847b9b3d3-config-volume\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820490 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820509 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820524 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820541 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820582 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-tls\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820631 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d991975-b491-4d47-9345-1c021afcb155-crio-socket\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820679 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944dd579-cc50-401c-a32f-04055024d5f8-serviceca\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820680 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1294b06a-d192-4f92-bade-e90847b9b3d3-config-volume\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820710 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820678 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820730 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820748 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820787 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-kubelet\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820817 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnb9\" (UniqueName: \"kubernetes.io/projected/80f091be-29ea-430b-a509-e24f1f3169dc-kube-api-access-whnb9\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:06:42.820832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820838 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820836 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-tls\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820854 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-var-lib-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820877 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-bin\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.820901 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821029 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d991975-b491-4d47-9345-1c021afcb155-data-volume\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821117 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821137 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821154 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294b06a-d192-4f92-bade-e90847b9b3d3-tmp-dir\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821117 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d991975-b491-4d47-9345-1c021afcb155-data-volume\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821194 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-script-lib\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821260 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1294b06a-d192-4f92-bade-e90847b9b3d3-tmp-dir\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821321 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821326 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7k9\" (UniqueName: \"kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821356 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821376 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821390 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-script-lib\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821395 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821430 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj66d\" (UniqueName: \"kubernetes.io/projected/62a68ba4-c84a-47ea-812e-0730bec1645e-kube-api-access-nj66d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821450 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821455 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821488 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821517 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821545 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:42.821603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821574 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-run\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821693 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821707 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-netns\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821725 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715adb78-caaa-4665-a368-66bd25143c59-metrics-certs\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821762 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821782 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821808 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821828 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-kubelet-config\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821844 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wxtll\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821859 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821845 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68679646-df21-499f-b5dd-c5b6ceb12314-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821914 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821922 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821917 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prddx\" (UniqueName: \"kubernetes.io/projected/946e5a1e-bfdd-48e2-81a5-cb512473ef85-kube-api-access-prddx\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821976 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68679646-df21-499f-b5dd-c5b6ceb12314-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821968 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821924 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821989 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.821916 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822003 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysconfig\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822051 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-sys\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822087 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-accelerators-collector-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822115 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-daemon-config\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822138 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-socket-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822155 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822170 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-lib-modules\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822286 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822283 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-accelerators-collector-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822354 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-systemd-units\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.822442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822390 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/275f0ed9-b587-4bda-996c-56f53df75d68-ovn-node-metrics-cert\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822492 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822512 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822530 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-kubernetes\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822553 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822570 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-daemon-config\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822585 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822596 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/275f0ed9-b587-4bda-996c-56f53df75d68-ovn-node-metrics-cert\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822606 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-binary-copy\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822625 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822650 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822703 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cnibin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822791 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822798 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-cni-binary-copy\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822876 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822968 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9tsd7\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822976 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.822972 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823014 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823019 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823020 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823040 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-multus\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823049 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823074 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823088 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xbrch\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823099 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmnro8141ghht\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823087 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823162 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823240 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823283 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0ef628-8414-4224-82df-4cede1a6d05b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823307 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823337 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-hostroot\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823364 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-cnibin\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823390 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2bm\" (UniqueName: \"kubernetes.io/projected/1294b06a-d192-4f92-bade-e90847b9b3d3-kube-api-access-7q2bm\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823419 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-conf\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823428 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823443 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sggrc\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823446 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823460 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823500 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8wz\" (UniqueName: \"kubernetes.io/projected/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-kube-api-access-cw8wz\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823498 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823512 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823528 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/80f091be-29ea-430b-a509-e24f1f3169dc-iptables-alerter-script\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823531 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823551 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823616 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9ztjt\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823622 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823422 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823655 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823658 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823668 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.823655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823686 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-multus-certs\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823706 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823815 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-system-cni-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823932 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-kubelet\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.823977 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824010 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cxtc9\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824017 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824029 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-etc-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824096 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-config\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824147 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn862\" (UniqueName: \"kubernetes.io/projected/8d991975-b491-4d47-9345-1c021afcb155-kube-api-access-vn862\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824306 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824339 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824368 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824377 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-ovnkube-config\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824383 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824390 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/80f091be-29ea-430b-a509-e24f1f3169dc-iptables-alerter-script\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824312 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824434 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824395 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-system-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824482 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824531 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-netd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824593 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824484 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824696 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824382 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4015660-eec1-47d8-8432-de3b74803f6a-original-pull-secret\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824685 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d991975-b491-4d47-9345-1c021afcb155-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824848 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824851 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-registration-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824877 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-host\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.824902 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-root\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.825016 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d991975-b491-4d47-9345-1c021afcb155-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.825061 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.825562 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 08:06:42.825765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.825699 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 08:06:42.827163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.826544 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 08:06:42.827163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.826930 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.829391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829360 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/275f0ed9-b587-4bda-996c-56f53df75d68-env-overrides\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829766 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829742 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829814 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829823 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829734 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829753 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829851 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1294b06a-d192-4f92-bade-e90847b9b3d3-metrics-tls\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829845 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d991975-b491-4d47-9345-1c021afcb155-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829875 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829894 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.830016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.829980 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.830546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.830418 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.830839 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.830817 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.830927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.830864 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.830927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.830885 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.830927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.830915 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.831025 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.830939 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.831226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.831212 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.831328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.831317 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.832292 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.832277 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.832354 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.832298 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.832416 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.832395 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:42.832469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.832436 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.832543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.832527 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.832543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.832539 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/68679646-df21-499f-b5dd-c5b6ceb12314-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:42.832741 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.832727 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.837684 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.837664 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.869697 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.869674 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6l7h\" (UniqueName: \"kubernetes.io/projected/715adb78-caaa-4665-a368-66bd25143c59-kube-api-access-p6l7h\") pod \"network-metrics-daemon-zjzfb\" (UID: \"715adb78-caaa-4665-a368-66bd25143c59\") " pod="openshift-multus/network-metrics-daemon-zjzfb" Nov 02 08:06:42.870081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.870062 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzpcr\" (UniqueName: \"kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr\") pod \"console-5dd89b74f-67l4d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:42.870164 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.870147 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzh8\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8\") pod \"alertmanager-main-0\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:06:42.870278 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.870263 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48h5d\" (UniqueName: \"kubernetes.io/projected/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-kube-api-access-48h5d\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:06:42.870584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.870569 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4qls\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls\") pod \"prometheus-k8s-0\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:42.871307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.871285 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77fv\" (UniqueName: \"kubernetes.io/projected/944dd579-cc50-401c-a32f-04055024d5f8-kube-api-access-d77fv\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:06:42.871385 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.871371 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28h6\" (UniqueName: \"kubernetes.io/projected/275f0ed9-b587-4bda-996c-56f53df75d68-kube-api-access-b28h6\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.871484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.871470 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtv7l\" (UniqueName: \"kubernetes.io/projected/b5f83ac7-e0a5-47db-bf75-f72a32394313-kube-api-access-xtv7l\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.871527 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.871472 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwq8d\" (UniqueName: \"kubernetes.io/projected/68679646-df21-499f-b5dd-c5b6ceb12314-kube-api-access-kwq8d\") pod \"prometheus-operator-9cd89b74c-bpdm6\" (UID: \"68679646-df21-499f-b5dd-c5b6ceb12314\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bpdm6" Nov 02 08:06:42.871527 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.871514 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbtvc\" (UniqueName: \"kubernetes.io/projected/edbc19ec-63f1-446d-989e-97fdc12575f7-kube-api-access-rbtvc\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.872434 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.872419 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnb9\" (UniqueName: \"kubernetes.io/projected/80f091be-29ea-430b-a509-e24f1f3169dc-kube-api-access-whnb9\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:06:42.872633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.872617 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqw98\" (UniqueName: \"kubernetes.io/projected/ee0ef628-8414-4224-82df-4cede1a6d05b-kube-api-access-jqw98\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.874113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.874092 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj66d\" (UniqueName: \"kubernetes.io/projected/62a68ba4-c84a-47ea-812e-0730bec1645e-kube-api-access-nj66d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.874203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.874124 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/a55d1dbd-a295-4b42-a0fb-b16022237e8c-kube-api-access-2kv9r\") pod \"network-check-target-rshcn\" (UID: \"a55d1dbd-a295-4b42-a0fb-b16022237e8c\") " pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:42.874240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.874229 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prddx\" (UniqueName: \"kubernetes.io/projected/946e5a1e-bfdd-48e2-81a5-cb512473ef85-kube-api-access-prddx\") pod \"ingress-canary-jddzv\" (UID: \"946e5a1e-bfdd-48e2-81a5-cb512473ef85\") " pod="openshift-ingress-canary/ingress-canary-jddzv" Nov 02 08:06:42.874501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.874486 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7k9\" (UniqueName: \"kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9\") pod \"console-6bcd5f847b-zrw55\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:42.874654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.874638 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2bm\" (UniqueName: \"kubernetes.io/projected/1294b06a-d192-4f92-bade-e90847b9b3d3-kube-api-access-7q2bm\") pod \"dns-default-l7nzr\" (UID: \"1294b06a-d192-4f92-bade-e90847b9b3d3\") " pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:42.874714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.874670 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8wz\" (UniqueName: \"kubernetes.io/projected/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-kube-api-access-cw8wz\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.875073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.875053 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn862\" (UniqueName: \"kubernetes.io/projected/8d991975-b491-4d47-9345-1c021afcb155-kube-api-access-vn862\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.925358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925328 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-multus-certs\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.925358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925367 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-system-cni-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925393 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-kubelet\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925458 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-kubelet\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925465 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-system-cni-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925485 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-etc-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925525 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-system-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925534 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-etc-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925550 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-netd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925576 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-registration-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925586 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-netd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925465 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-multus-certs\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925600 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-host\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925625 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-root\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925648 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-host\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925653 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80f091be-29ea-430b-a509-e24f1f3169dc-host-slash\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925643 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-registration-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925677 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-sys\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925695 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80f091be-29ea-430b-a509-e24f1f3169dc-host-slash\") pod \"iptables-alerter-5g226\" (UID: \"80f091be-29ea-430b-a509-e24f1f3169dc\") " pod="openshift-network-operator/iptables-alerter-5g226" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925711 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-log-socket\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925603 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-system-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.925711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925706 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-sys\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925669 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-root\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925735 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-dbus\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925744 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-log-socket\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925764 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-bin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925784 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-dbus\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925797 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-socket-dir-parent\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925804 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-bin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925824 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-device-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925849 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-socket-dir-parent\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925859 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-node-log\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925868 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-device-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925889 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-netns\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925911 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-k8s-cni-cncf-io\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925926 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-ovn\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925938 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-netns\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925944 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-sys-fs\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925968 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-slash\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925972 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-ovn\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925977 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-node-log\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925972 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-run-k8s-cni-cncf-io\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.925996 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926004 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-sys-fs\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926016 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-slash\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926024 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926025 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-modprobe-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926058 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-systemd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926077 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-systemd\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926091 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-var-lib-kubelet\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926109 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926120 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-systemd\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926123 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-modprobe-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926131 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944dd579-cc50-401c-a32f-04055024d5f8-host\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926142 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-systemd\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926154 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-etc-kubernetes\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926162 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926162 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-var-lib-kubelet\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926189 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926163 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944dd579-cc50-401c-a32f-04055024d5f8-host\") pod \"node-ca-8v9wz\" (UID: \"944dd579-cc50-401c-a32f-04055024d5f8\") " pod="openshift-image-registry/node-ca-8v9wz" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926209 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-etc-kubernetes\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926228 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-wtmp\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926257 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-conf-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926288 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-hosts-file\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926325 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926327 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d54dbca4-a028-4ec4-8aae-6edd83aaff7b-hosts-file\") pod \"node-resolver-ndqtq\" (UID: \"d54dbca4-a028-4ec4-8aae-6edd83aaff7b\") " pod="openshift-dns/node-resolver-ndqtq" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926291 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-conf-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926342 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d991975-b491-4d47-9345-1c021afcb155-crio-socket\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926346 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-d\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926324 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-node-exporter-wtmp\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926358 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-run-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926358 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-kubelet\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926381 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-kubelet\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926389 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-var-lib-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926416 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-bin\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926365 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d991975-b491-4d47-9345-1c021afcb155-crio-socket\") pod \"insights-runtime-extractor-kck72\" (UID: \"8d991975-b491-4d47-9345-1c021afcb155\") " pod="openshift-insights/insights-runtime-extractor-kck72" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926447 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-run\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926470 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-netns\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926476 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-cni-bin\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926498 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-kubelet-config\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926507 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-run\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926471 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-var-lib-openvswitch\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926531 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-netns\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926539 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysconfig\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926548 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4015660-eec1-47d8-8432-de3b74803f6a-kubelet-config\") pod \"global-pull-secret-syncer-xs697\" (UID: \"f4015660-eec1-47d8-8432-de3b74803f6a\") " pod="kube-system/global-pull-secret-syncer-xs697" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926573 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-sys\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926591 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-socket-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926608 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-lib-modules\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926615 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5f83ac7-e0a5-47db-bf75-f72a32394313-sys\") pod \"node-exporter-m89dq\" (UID: \"b5f83ac7-e0a5-47db-bf75-f72a32394313\") " pod="openshift-monitoring/node-exporter-m89dq" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926605 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysconfig\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926629 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-systemd-units\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926641 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-socket-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926654 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926657 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-systemd-units\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926671 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-kubernetes\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926686 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926691 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926702 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-lib-modules\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926710 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cnibin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926731 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/edbc19ec-63f1-446d-989e-97fdc12575f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n48sm\" (UID: \"edbc19ec-63f1-446d-989e-97fdc12575f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n48sm" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926734 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-kubernetes\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.926651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926735 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926769 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275f0ed9-b587-4bda-996c-56f53df75d68-host-run-ovn-kubernetes\") pod \"ovnkube-node-2r44q\" (UID: \"275f0ed9-b587-4bda-996c-56f53df75d68\") " pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926782 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926764 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-cnibin\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926805 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-multus\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926821 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-hostroot\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926834 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-cnibin\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926851 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-conf\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926870 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-host-var-lib-cni-multus\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926881 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-hostroot\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926885 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee0ef628-8414-4224-82df-4cede1a6d05b-cnibin\") pod \"multus-additional-cni-plugins-82pzl\" (UID: \"ee0ef628-8414-4224-82df-4cede1a6d05b\") " pod="openshift-multus/multus-additional-cni-plugins-82pzl" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926906 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d71abbe8-6a3a-4d0f-9a55-6fe051cf0017-multus-cni-dir\") pod \"multus-rclzx\" (UID: \"d71abbe8-6a3a-4d0f-9a55-6fe051cf0017\") " pod="openshift-multus/multus-rclzx" Nov 02 08:06:42.929088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.926982 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62a68ba4-c84a-47ea-812e-0730bec1645e-etc-sysctl-conf\") pod \"tuned-ppnkn\" (UID: \"62a68ba4-c84a-47ea-812e-0730bec1645e\") " pod="openshift-cluster-node-tuning-operator/tuned-ppnkn" Nov 02 08:06:42.976000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:42.975982 6160 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:06:43.124043 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:43.123956 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:43.318588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:43.318550 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:43.479209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:43.479153 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:43.544462 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:43.544426 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcd5f847b-zrw55"] Nov 02 08:06:44.622790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:44.622757 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l7nzr" Nov 02 08:06:44.667791 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:44.667764 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerStarted","Data":"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607"} Nov 02 08:06:44.667987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:44.667828 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerStarted","Data":"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224"} Nov 02 08:06:44.667987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:44.667856 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerStarted","Data":"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5"} Nov 02 08:06:44.667987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:44.667880 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerStarted","Data":"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e"} Nov 02 08:06:45.131932 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:45.131891 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:45.136417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:45.136395 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:45.452633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:45.452599 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:45.477460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:45.477429 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:45.684271 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:45.684226 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:46.332126 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:46.332095 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:46.381991 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:46.381962 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:46.484493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:46.484450 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:47.286448 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:47.286407 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:47.286904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:47.286614 6160 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:06:47.305485 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:47.305450 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:06:47.488975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:47.488943 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:47.700322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:47.700285 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:48.163909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:48.163878 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:06:49.465795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:49.465758 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:06:49.640652 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:49.640627 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:06:49.640879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:49.640864 6160 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 08:06:50.675872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:50.675782 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:50.828264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:50.828234 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:55.482113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:55.482083 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:06:56.353799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:56.353752 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:56.611483 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:56.611395 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:57.332601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:06:57.332573 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2r44q" Nov 02 08:07:02.052194 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:02.052139 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:02.197528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:02.197489 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:04.438213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:04.438163 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7jfqx" Nov 02 08:07:05.663782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:05.663740 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:05.851928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:05.851888 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:08.677613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:08.677572 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6bcd5f847b-zrw55" podUID="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" containerName="console" containerID="cri-o://207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07" gracePeriod=15 Nov 02 08:07:08.929023 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:08.928964 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:07:09.034935 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.034891 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert\") pod \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " Nov 02 08:07:09.034935 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.034940 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca\") pod \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " Nov 02 08:07:09.035211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.034974 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp\") pod \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " Nov 02 08:07:09.035211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035091 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config\") pod \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " Nov 02 08:07:09.035211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035150 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle\") pod \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " Nov 02 08:07:09.035211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035204 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9\" (UniqueName: \"kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9\") pod \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " Nov 02 08:07:09.035433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035248 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert\") pod \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " Nov 02 08:07:09.035433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035256 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp" pod "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" (UID: "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:07:09.035433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035305 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config\") pod \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\" (UID: \"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c\") " Nov 02 08:07:09.035591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035468 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca" pod "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" (UID: "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:09.035937 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035906 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config" pod "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" (UID: "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:09.036606 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.036131 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle" pod "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" (UID: "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:09.036748 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035463 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert" pod "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" (UID: "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:09.036838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.035925 6160 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:09.036947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.036934 6160 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-tmp\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:09.044516 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.044466 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert" pod "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" (UID: "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:09.044646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.044608 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config" pod "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" (UID: "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:09.044704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.044658 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9" pod "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" (UID: "bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c"). InnerVolumeSpecName "kube-api-access-9s7k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:07:09.138111 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.138079 6160 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-trusted-ca-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:09.138111 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.138109 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s7k9\" (UniqueName: \"kubernetes.io/projected/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-kube-api-access-9s7k9\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:09.138336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.138120 6160 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-serving-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:09.138336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.138141 6160 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-oauth-config\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:09.138336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.138151 6160 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-oauth-serving-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:09.138336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.138162 6160 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c-console-config\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:09.341987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.341899 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 08:07:09.342331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.342280 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="alertmanager" containerID="cri-o://325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f" gracePeriod=120 Nov 02 08:07:09.342489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.342365 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy" containerID="cri-o://f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b" gracePeriod=120 Nov 02 08:07:09.342489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.342386 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="config-reloader" containerID="cri-o://6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607" gracePeriod=120 Nov 02 08:07:09.342489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.342408 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="prom-label-proxy" containerID="cri-o://f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc" gracePeriod=120 Nov 02 08:07:09.342489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.342404 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy-web" containerID="cri-o://21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df" gracePeriod=120 Nov 02 08:07:09.342711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.342357 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy-metric" containerID="cri-o://fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d" gracePeriod=120 Nov 02 08:07:09.419803 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.419769 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd5dcfd5d-zhfd5"] Nov 02 08:07:09.420075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.420062 6160 cpu_manager.go:414] "containerMap: removing container" podUID="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" containerName="console" Nov 02 08:07:09.420116 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.420077 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" containerName="console" Nov 02 08:07:09.420198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.420186 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" containerName="console" Nov 02 08:07:09.421951 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.421935 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.469213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.469168 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rshcn" Nov 02 08:07:09.495548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.495516 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd5dcfd5d-zhfd5"] Nov 02 08:07:09.540423 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.540388 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-trusted-ca-bundle\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.540423 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.540425 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c925d891-89c5-4853-9df9-f0114e2c093c-tmp\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.540640 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.540562 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-console-config\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.540640 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.540617 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-service-ca\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.540787 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.540765 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-oauth-serving-cert\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.540821 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.540799 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c925d891-89c5-4853-9df9-f0114e2c093c-console-oauth-config\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.540921 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.540900 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c925d891-89c5-4853-9df9-f0114e2c093c-console-serving-cert\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.540970 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.540940 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdf8\" (UniqueName: \"kubernetes.io/projected/c925d891-89c5-4853-9df9-f0114e2c093c-kube-api-access-9hdf8\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.641785 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.641706 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-console-config\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.641785 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.641760 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-service-ca\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.641785 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-oauth-serving-cert\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.641811 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c925d891-89c5-4853-9df9-f0114e2c093c-console-oauth-config\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.641838 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c925d891-89c5-4853-9df9-f0114e2c093c-console-serving-cert\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.641956 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdf8\" (UniqueName: \"kubernetes.io/projected/c925d891-89c5-4853-9df9-f0114e2c093c-kube-api-access-9hdf8\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.642033 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-trusted-ca-bundle\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.642067 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c925d891-89c5-4853-9df9-f0114e2c093c-tmp\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.642235 6160 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 08:07:09.642514 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.642467 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c925d891-89c5-4853-9df9-f0114e2c093c-tmp\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642628 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.642613 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-service-ca\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.642701 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-oauth-serving-cert\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.642992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.642970 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-console-config\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.643481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.643457 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c925d891-89c5-4853-9df9-f0114e2c093c-trusted-ca-bundle\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.645102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.645079 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c925d891-89c5-4853-9df9-f0114e2c093c-console-oauth-config\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.645251 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.645236 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c925d891-89c5-4853-9df9-f0114e2c093c-console-serving-cert\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.696041 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.696011 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdf8\" (UniqueName: \"kubernetes.io/projected/c925d891-89c5-4853-9df9-f0114e2c093c-kube-api-access-9hdf8\") pod \"console-fd5dcfd5d-zhfd5\" (UID: \"c925d891-89c5-4853-9df9-f0114e2c093c\") " pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.728627 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.728590 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:09.732395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.732372 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:09.756498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756468 6160 generic.go:356] "Generic (PLEG): container finished" podID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerID="f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc" exitCode=0 Nov 02 08:07:09.756498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756494 6160 generic.go:356] "Generic (PLEG): container finished" podID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerID="fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d" exitCode=0 Nov 02 08:07:09.756658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756503 6160 generic.go:356] "Generic (PLEG): container finished" podID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerID="f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b" exitCode=0 Nov 02 08:07:09.756658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756514 6160 generic.go:356] "Generic (PLEG): container finished" podID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerID="6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607" exitCode=0 Nov 02 08:07:09.756658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756510 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerDied","Data":"f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc"} Nov 02 08:07:09.756658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756522 6160 generic.go:356] "Generic (PLEG): container finished" podID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerID="325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f" exitCode=0 Nov 02 08:07:09.756658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756586 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerDied","Data":"fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d"} Nov 02 08:07:09.756658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756625 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerDied","Data":"f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b"} Nov 02 08:07:09.756658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756659 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerDied","Data":"6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607"} Nov 02 08:07:09.756960 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.756689 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerDied","Data":"325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f"} Nov 02 08:07:09.757961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.757934 6160 generic.go:356] "Generic (PLEG): container finished" podID="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" containerID="207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07" exitCode=0 Nov 02 08:07:09.758078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.758023 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5f847b-zrw55" event={"ID":"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c","Type":"ContainerDied","Data":"207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07"} Nov 02 08:07:09.758078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.758063 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5f847b-zrw55" event={"ID":"bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c","Type":"ContainerDied","Data":"45f95867ea5f3f86697e78995bc373467e88e894eb245f262be2e65deaf2c9a3"} Nov 02 08:07:09.758078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.758070 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5f847b-zrw55" Nov 02 08:07:09.758256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.758105 6160 scope.go:122] "RemoveContainer" containerID="207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07" Nov 02 08:07:09.766806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.766789 6160 scope.go:122] "RemoveContainer" containerID="207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07" Nov 02 08:07:09.767126 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:09.767097 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07\": container with ID starting with 207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07 not found: ID does not exist" containerID="207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07" Nov 02 08:07:09.767214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.767141 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07"} err="failed to get container status \"207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07\": rpc error: code = NotFound desc = could not find container \"207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07\": container with ID starting with 207d2b0bb14833b3a4e27fa4115495b1eaff08047cfeabe6de6eb1feee32ef07 not found: ID does not exist" Nov 02 08:07:09.909799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.909712 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcd5f847b-zrw55"] Nov 02 08:07:09.916018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.915995 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bcd5f847b-zrw55"] Nov 02 08:07:09.936618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:09.936583 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:10.098410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.098379 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd5dcfd5d-zhfd5"] Nov 02 08:07:10.101703 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:07:10.101674 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc925d891_89c5_4853_9df9_f0114e2c093c.slice/crio-f2877b62200bbb1f829163665c3029fc594a6b79ace78123cc215fa122800665 WatchSource:0}: Error finding container f2877b62200bbb1f829163665c3029fc594a6b79ace78123cc215fa122800665: Status 404 returned error can't find the container with id f2877b62200bbb1f829163665c3029fc594a6b79ace78123cc215fa122800665 Nov 02 08:07:10.124773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.124742 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:10.271075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.271040 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:10.521619 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:10.521537 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:07:10.592894 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.592871 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:07:10.650509 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.650480 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.650663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.650527 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.650663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.650556 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.650663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.650585 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.650663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.650604 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.650663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.650621 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.650663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.650644 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.650986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.650943 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.651040 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.651007 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.651093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.651070 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.651144 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.651091 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:10.651144 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.651105 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.651439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.651209 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.651439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.651246 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config\") pod \"5a387bc4-fe09-4921-9ff4-8199642acb38\" (UID: \"5a387bc4-fe09-4921-9ff4-8199642acb38\") " Nov 02 08:07:10.652303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.651532 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:07:10.652607 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.651621 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:10.652735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.652478 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-metrics-client-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.654037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.654011 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:10.654147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.654069 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:10.654305 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.654286 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:07:10.654411 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.654339 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "kube-api-access-5jzh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:07:10.654486 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.654429 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:07:10.654950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.654931 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:10.655214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.655151 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:10.655444 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.655421 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:10.658740 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.658721 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:10.664365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.664343 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config" pod "5a387bc4-fe09-4921-9ff4-8199642acb38" (UID: "5a387bc4-fe09-4921-9ff4-8199642acb38"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:10.753717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753685 6160 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.753717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753713 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jzh8\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-kube-api-access-5jzh8\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753728 6160 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753741 6160 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-web-config\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753750 6160 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753758 6160 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-config-out\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753767 6160 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-main-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753777 6160 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753787 6160 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a387bc4-fe09-4921-9ff4-8199642acb38-alertmanager-main-db\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753796 6160 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753810 6160 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a387bc4-fe09-4921-9ff4-8199642acb38-cluster-tls-config\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.754109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.753820 6160 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a387bc4-fe09-4921-9ff4-8199642acb38-tls-assets\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:10.764717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.764686 6160 generic.go:356] "Generic (PLEG): container finished" podID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerID="21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df" exitCode=0 Nov 02 08:07:10.764879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.764768 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 08:07:10.764879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.764767 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerDied","Data":"21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df"} Nov 02 08:07:10.764879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.764850 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a387bc4-fe09-4921-9ff4-8199642acb38","Type":"ContainerDied","Data":"8635ab5968e926289e2ff01e43c2988edbc62ef85663db5b03d7721ca42b8b5a"} Nov 02 08:07:10.765028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.764899 6160 scope.go:122] "RemoveContainer" containerID="f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc" Nov 02 08:07:10.767779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.767746 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd5dcfd5d-zhfd5" event={"ID":"c925d891-89c5-4853-9df9-f0114e2c093c","Type":"ContainerStarted","Data":"2da401f49f48df2c4b824dbb639f67e00cbae668278de1344e0ea77e11f9de0a"} Nov 02 08:07:10.767882 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.767791 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd5dcfd5d-zhfd5" event={"ID":"c925d891-89c5-4853-9df9-f0114e2c093c","Type":"ContainerStarted","Data":"f2877b62200bbb1f829163665c3029fc594a6b79ace78123cc215fa122800665"} Nov 02 08:07:10.772893 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.772879 6160 scope.go:122] "RemoveContainer" containerID="fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d" Nov 02 08:07:10.779729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.779708 6160 scope.go:122] "RemoveContainer" containerID="f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b" Nov 02 08:07:10.786367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.786353 6160 scope.go:122] "RemoveContainer" containerID="21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df" Nov 02 08:07:10.792819 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.792803 6160 scope.go:122] "RemoveContainer" containerID="6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607" Nov 02 08:07:10.800927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.800392 6160 scope.go:122] "RemoveContainer" containerID="325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f" Nov 02 08:07:10.809637 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.809617 6160 scope.go:122] "RemoveContainer" containerID="a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be" Nov 02 08:07:10.819062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.819046 6160 scope.go:122] "RemoveContainer" containerID="f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc" Nov 02 08:07:10.819366 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:10.819349 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc\": container with ID starting with f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc not found: ID does not exist" containerID="f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc" Nov 02 08:07:10.819449 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.819376 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc"} err="failed to get container status \"f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc\": rpc error: code = NotFound desc = could not find container \"f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc\": container with ID starting with f459d7984a4842e00d2e90a4ad5b589fc10303935edfc5f694ee1c28b62a62fc not found: ID does not exist" Nov 02 08:07:10.819449 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.819400 6160 scope.go:122] "RemoveContainer" containerID="fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d" Nov 02 08:07:10.819668 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:10.819644 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d\": container with ID starting with fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d not found: ID does not exist" containerID="fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d" Nov 02 08:07:10.819778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.819675 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d"} err="failed to get container status \"fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d\": rpc error: code = NotFound desc = could not find container \"fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d\": container with ID starting with fecd6dbe7bf7019099dff02f7aabf0172551598ff271651cb91e941ba030826d not found: ID does not exist" Nov 02 08:07:10.819778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.819697 6160 scope.go:122] "RemoveContainer" containerID="f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b" Nov 02 08:07:10.819967 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:10.819943 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b\": container with ID starting with f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b not found: ID does not exist" containerID="f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b" Nov 02 08:07:10.820033 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.819975 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b"} err="failed to get container status \"f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b\": rpc error: code = NotFound desc = could not find container \"f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b\": container with ID starting with f3e6b26393cd1561d509ebd1cad189793d1bcb71898c3150b1f2b0d2c355e23b not found: ID does not exist" Nov 02 08:07:10.820033 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.819997 6160 scope.go:122] "RemoveContainer" containerID="21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df" Nov 02 08:07:10.820363 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:10.820344 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df\": container with ID starting with 21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df not found: ID does not exist" containerID="21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df" Nov 02 08:07:10.820439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.820366 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df"} err="failed to get container status \"21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df\": rpc error: code = NotFound desc = could not find container \"21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df\": container with ID starting with 21dbedcdf1717cff8c7fb9d0f91dcd0bbc81f7c2d260b641ef9970ad8f5ec5df not found: ID does not exist" Nov 02 08:07:10.820439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.820380 6160 scope.go:122] "RemoveContainer" containerID="6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607" Nov 02 08:07:10.820633 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:10.820614 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607\": container with ID starting with 6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607 not found: ID does not exist" containerID="6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607" Nov 02 08:07:10.820683 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.820637 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607"} err="failed to get container status \"6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607\": rpc error: code = NotFound desc = could not find container \"6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607\": container with ID starting with 6820fe644280b331babd80541334f81c757544b85f0a4e6a841caeb28cffc607 not found: ID does not exist" Nov 02 08:07:10.820683 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.820651 6160 scope.go:122] "RemoveContainer" containerID="325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f" Nov 02 08:07:10.820966 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:10.820943 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f\": container with ID starting with 325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f not found: ID does not exist" containerID="325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f" Nov 02 08:07:10.821016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.820975 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f"} err="failed to get container status \"325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f\": rpc error: code = NotFound desc = could not find container \"325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f\": container with ID starting with 325e5d62e0d1f46e3978cc6d135f0b3cd6f23fb1945acd5499a149dad123138f not found: ID does not exist" Nov 02 08:07:10.821016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.820997 6160 scope.go:122] "RemoveContainer" containerID="a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be" Nov 02 08:07:10.821323 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:10.821297 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be\": container with ID starting with a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be not found: ID does not exist" containerID="a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be" Nov 02 08:07:10.821391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.821322 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be"} err="failed to get container status \"a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be\": rpc error: code = NotFound desc = could not find container \"a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be\": container with ID starting with a0c83f01b004ad9fbe716690986f0087392aa330099a737e4046fa2125e726be not found: ID does not exist" Nov 02 08:07:10.922455 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:10.922395 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-fd5dcfd5d-zhfd5" podStartSLOduration=1.922377161 podStartE2EDuration="1.922377161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:07:10.920743474 +0000 UTC m=+29.474247671" watchObservedRunningTime="2026-11-02 08:07:10.922377161 +0000 UTC m=+29.475881358" Nov 02 08:07:11.025742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:11.025571 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 08:07:11.028791 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:11.028763 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 08:07:11.581888 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:11.581854 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" path="/var/lib/kubelet/pods/5a387bc4-fe09-4921-9ff4-8199642acb38/volumes" Nov 02 08:07:11.582557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:11.582542 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c" path="/var/lib/kubelet/pods/bcd4a3e6-5543-48a2-8cdf-faa3b33ca11c/volumes" Nov 02 08:07:12.182447 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:12.182413 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:15.037775 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.037737 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 08:07:15.038693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.038633 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="prometheus" containerID="cri-o://3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" gracePeriod=600 Nov 02 08:07:15.038867 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.038828 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" gracePeriod=600 Nov 02 08:07:15.038993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.038935 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="thanos-sidecar" containerID="cri-o://d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" gracePeriod=600 Nov 02 08:07:15.039091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.039062 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy" containerID="cri-o://be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" gracePeriod=600 Nov 02 08:07:15.039150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.039135 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy-web" containerID="cri-o://f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" gracePeriod=600 Nov 02 08:07:15.039260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.038939 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="config-reloader" containerID="cri-o://e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" gracePeriod=600 Nov 02 08:07:15.283148 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.283124 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:15.391131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391046 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391086 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391105 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391124 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391446 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391148 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391446 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391339 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391446 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391402 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391448 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391483 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391513 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391545 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391587 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391605 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:15.391858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391622 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.391858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391669 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.392024 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.391938 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:15.392493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.392461 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:15.392615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.392503 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:15.392615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.392522 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.392615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.392557 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.392615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.392604 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.392827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.392639 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.392827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.392681 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.392827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.392710 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy\") pod \"40a4044b-18d1-494c-b50e-2b3d253efdf2\" (UID: \"40a4044b-18d1-494c-b50e-2b3d253efdf2\") " Nov 02 08:07:15.392997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.392971 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:15.393339 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.393316 6160 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.393619 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.393592 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:15.393816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.393345 6160 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-metrics-client-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.393892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.393831 6160 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.393892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.393849 6160 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.393892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.393867 6160 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.394539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.394514 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:07:15.394759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.394733 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:15.395774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.395734 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:15.395872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.395815 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:15.396159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.396125 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:15.396159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.396142 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:07:15.396336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.396159 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:15.396403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.396366 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:07:15.396451 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.396411 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:15.396451 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.396432 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "kube-api-access-s4qls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:07:15.396557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.396484 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:15.396822 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.396805 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:15.397936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.397917 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:15.407796 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.407768 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config" pod "40a4044b-18d1-494c-b50e-2b3d253efdf2" (UID: "40a4044b-18d1-494c-b50e-2b3d253efdf2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:15.494501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494463 6160 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494492 6160 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-config\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494502 6160 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494511 6160 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494521 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4qls\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-kube-api-access-s4qls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494530 6160 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-db\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494539 6160 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40a4044b-18d1-494c-b50e-2b3d253efdf2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494549 6160 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494561 6160 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-web-config\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494569 6160 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-kube-rbac-proxy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494579 6160 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494587 6160 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40a4044b-18d1-494c-b50e-2b3d253efdf2-tls-assets\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494596 6160 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-grpc-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494606 6160 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40a4044b-18d1-494c-b50e-2b3d253efdf2-secret-metrics-client-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.494761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.494615 6160 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40a4044b-18d1-494c-b50e-2b3d253efdf2-config-out\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:15.788881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.788837 6160 generic.go:356] "Generic (PLEG): container finished" podID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerID="7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" exitCode=0 Nov 02 08:07:15.788881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.788866 6160 generic.go:356] "Generic (PLEG): container finished" podID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerID="be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" exitCode=0 Nov 02 08:07:15.788881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.788873 6160 generic.go:356] "Generic (PLEG): container finished" podID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerID="f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" exitCode=0 Nov 02 08:07:15.789153 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.788892 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerDied","Data":"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607"} Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789153 6160 generic.go:356] "Generic (PLEG): container finished" podID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerID="d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" exitCode=0 Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789314 6160 generic.go:356] "Generic (PLEG): container finished" podID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerID="e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" exitCode=0 Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789313 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerDied","Data":"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224"} Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789333 6160 generic.go:356] "Generic (PLEG): container finished" podID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerID="3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" exitCode=0 Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789432 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerDied","Data":"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5"} Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789507 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerDied","Data":"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e"} Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789511 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789550 6160 scope.go:122] "RemoveContainer" containerID="7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789822 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerDied","Data":"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7"} Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789886 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerDied","Data":"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568"} Nov 02 08:07:15.793088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.789942 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40a4044b-18d1-494c-b50e-2b3d253efdf2","Type":"ContainerDied","Data":"393a34c14ee09719bfdab0517a1b5cac495b200fd4fa3f6d8958e7e3ca608cad"} Nov 02 08:07:15.800723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.800706 6160 scope.go:122] "RemoveContainer" containerID="be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" Nov 02 08:07:15.807599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.807579 6160 scope.go:122] "RemoveContainer" containerID="f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" Nov 02 08:07:15.814054 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.814038 6160 scope.go:122] "RemoveContainer" containerID="d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" Nov 02 08:07:15.820381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.820355 6160 scope.go:122] "RemoveContainer" containerID="e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" Nov 02 08:07:15.826759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.826743 6160 scope.go:122] "RemoveContainer" containerID="3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" Nov 02 08:07:15.834019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.834001 6160 scope.go:122] "RemoveContainer" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" Nov 02 08:07:15.840248 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.840233 6160 scope.go:122] "RemoveContainer" containerID="7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" Nov 02 08:07:15.840485 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:15.840468 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": container with ID starting with 7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607 not found: ID does not exist" containerID="7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" Nov 02 08:07:15.840543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.840503 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607"} err="failed to get container status \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": rpc error: code = NotFound desc = could not find container \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": container with ID starting with 7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607 not found: ID does not exist" Nov 02 08:07:15.840543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.840520 6160 scope.go:122] "RemoveContainer" containerID="be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" Nov 02 08:07:15.840763 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:15.840741 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": container with ID starting with be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224 not found: ID does not exist" containerID="be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" Nov 02 08:07:15.840830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.840774 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224"} err="failed to get container status \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": rpc error: code = NotFound desc = could not find container \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": container with ID starting with be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224 not found: ID does not exist" Nov 02 08:07:15.840830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.840798 6160 scope.go:122] "RemoveContainer" containerID="f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" Nov 02 08:07:15.841036 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:15.841019 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": container with ID starting with f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5 not found: ID does not exist" containerID="f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" Nov 02 08:07:15.841070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841041 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5"} err="failed to get container status \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": rpc error: code = NotFound desc = could not find container \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": container with ID starting with f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5 not found: ID does not exist" Nov 02 08:07:15.841070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841054 6160 scope.go:122] "RemoveContainer" containerID="d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" Nov 02 08:07:15.841286 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:15.841270 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": container with ID starting with d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e not found: ID does not exist" containerID="d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" Nov 02 08:07:15.841335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841291 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e"} err="failed to get container status \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": rpc error: code = NotFound desc = could not find container \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": container with ID starting with d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e not found: ID does not exist" Nov 02 08:07:15.841335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841310 6160 scope.go:122] "RemoveContainer" containerID="e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" Nov 02 08:07:15.841523 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:15.841506 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": container with ID starting with e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7 not found: ID does not exist" containerID="e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" Nov 02 08:07:15.841585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841532 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7"} err="failed to get container status \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": rpc error: code = NotFound desc = could not find container \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": container with ID starting with e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7 not found: ID does not exist" Nov 02 08:07:15.841585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841552 6160 scope.go:122] "RemoveContainer" containerID="3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" Nov 02 08:07:15.841773 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:15.841759 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": container with ID starting with 3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568 not found: ID does not exist" containerID="3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" Nov 02 08:07:15.841809 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841775 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568"} err="failed to get container status \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": rpc error: code = NotFound desc = could not find container \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": container with ID starting with 3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568 not found: ID does not exist" Nov 02 08:07:15.841809 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841788 6160 scope.go:122] "RemoveContainer" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" Nov 02 08:07:15.841981 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:15.841965 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": container with ID starting with 1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2 not found: ID does not exist" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" Nov 02 08:07:15.842019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841985 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2"} err="failed to get container status \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": rpc error: code = NotFound desc = could not find container \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": container with ID starting with 1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2 not found: ID does not exist" Nov 02 08:07:15.842019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.841997 6160 scope.go:122] "RemoveContainer" containerID="7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" Nov 02 08:07:15.842206 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.842169 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607"} err="failed to get container status \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": rpc error: code = NotFound desc = could not find container \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": container with ID starting with 7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607 not found: ID does not exist" Nov 02 08:07:15.842206 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.842205 6160 scope.go:122] "RemoveContainer" containerID="be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" Nov 02 08:07:15.842409 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.842392 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224"} err="failed to get container status \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": rpc error: code = NotFound desc = could not find container \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": container with ID starting with be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224 not found: ID does not exist" Nov 02 08:07:15.842481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.842411 6160 scope.go:122] "RemoveContainer" containerID="f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" Nov 02 08:07:15.842612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.842594 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5"} err="failed to get container status \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": rpc error: code = NotFound desc = could not find container \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": container with ID starting with f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5 not found: ID does not exist" Nov 02 08:07:15.842657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.842614 6160 scope.go:122] "RemoveContainer" containerID="d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" Nov 02 08:07:15.842837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.842819 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e"} err="failed to get container status \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": rpc error: code = NotFound desc = could not find container \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": container with ID starting with d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e not found: ID does not exist" Nov 02 08:07:15.842909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.842837 6160 scope.go:122] "RemoveContainer" containerID="e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" Nov 02 08:07:15.843063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843046 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7"} err="failed to get container status \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": rpc error: code = NotFound desc = could not find container \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": container with ID starting with e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7 not found: ID does not exist" Nov 02 08:07:15.843115 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843064 6160 scope.go:122] "RemoveContainer" containerID="3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" Nov 02 08:07:15.843291 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843269 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568"} err="failed to get container status \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": rpc error: code = NotFound desc = could not find container \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": container with ID starting with 3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568 not found: ID does not exist" Nov 02 08:07:15.843358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843291 6160 scope.go:122] "RemoveContainer" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" Nov 02 08:07:15.843515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843499 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2"} err="failed to get container status \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": rpc error: code = NotFound desc = could not find container \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": container with ID starting with 1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2 not found: ID does not exist" Nov 02 08:07:15.843565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843515 6160 scope.go:122] "RemoveContainer" containerID="7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" Nov 02 08:07:15.843733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843708 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607"} err="failed to get container status \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": rpc error: code = NotFound desc = could not find container \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": container with ID starting with 7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607 not found: ID does not exist" Nov 02 08:07:15.843733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843728 6160 scope.go:122] "RemoveContainer" containerID="be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" Nov 02 08:07:15.843944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843926 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224"} err="failed to get container status \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": rpc error: code = NotFound desc = could not find container \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": container with ID starting with be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224 not found: ID does not exist" Nov 02 08:07:15.844010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.843945 6160 scope.go:122] "RemoveContainer" containerID="f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" Nov 02 08:07:15.844157 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.844142 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5"} err="failed to get container status \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": rpc error: code = NotFound desc = could not find container \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": container with ID starting with f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5 not found: ID does not exist" Nov 02 08:07:15.844213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.844157 6160 scope.go:122] "RemoveContainer" containerID="d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" Nov 02 08:07:15.844389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.844371 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e"} err="failed to get container status \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": rpc error: code = NotFound desc = could not find container \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": container with ID starting with d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e not found: ID does not exist" Nov 02 08:07:15.844465 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.844389 6160 scope.go:122] "RemoveContainer" containerID="e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" Nov 02 08:07:15.844596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.844580 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7"} err="failed to get container status \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": rpc error: code = NotFound desc = could not find container \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": container with ID starting with e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7 not found: ID does not exist" Nov 02 08:07:15.844638 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.844596 6160 scope.go:122] "RemoveContainer" containerID="3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" Nov 02 08:07:15.844831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.844813 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568"} err="failed to get container status \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": rpc error: code = NotFound desc = could not find container \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": container with ID starting with 3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568 not found: ID does not exist" Nov 02 08:07:15.844894 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.844832 6160 scope.go:122] "RemoveContainer" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" Nov 02 08:07:15.845081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845063 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2"} err="failed to get container status \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": rpc error: code = NotFound desc = could not find container \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": container with ID starting with 1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2 not found: ID does not exist" Nov 02 08:07:15.845081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845080 6160 scope.go:122] "RemoveContainer" containerID="7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" Nov 02 08:07:15.845331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845316 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607"} err="failed to get container status \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": rpc error: code = NotFound desc = could not find container \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": container with ID starting with 7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607 not found: ID does not exist" Nov 02 08:07:15.845378 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845331 6160 scope.go:122] "RemoveContainer" containerID="be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" Nov 02 08:07:15.845578 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845559 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224"} err="failed to get container status \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": rpc error: code = NotFound desc = could not find container \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": container with ID starting with be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224 not found: ID does not exist" Nov 02 08:07:15.845629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845579 6160 scope.go:122] "RemoveContainer" containerID="f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" Nov 02 08:07:15.845799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845785 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5"} err="failed to get container status \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": rpc error: code = NotFound desc = could not find container \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": container with ID starting with f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5 not found: ID does not exist" Nov 02 08:07:15.845833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845800 6160 scope.go:122] "RemoveContainer" containerID="d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" Nov 02 08:07:15.845989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845973 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e"} err="failed to get container status \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": rpc error: code = NotFound desc = could not find container \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": container with ID starting with d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e not found: ID does not exist" Nov 02 08:07:15.846025 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.845989 6160 scope.go:122] "RemoveContainer" containerID="e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" Nov 02 08:07:15.846220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.846203 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7"} err="failed to get container status \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": rpc error: code = NotFound desc = could not find container \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": container with ID starting with e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7 not found: ID does not exist" Nov 02 08:07:15.846294 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.846220 6160 scope.go:122] "RemoveContainer" containerID="3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" Nov 02 08:07:15.846438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.846420 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568"} err="failed to get container status \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": rpc error: code = NotFound desc = could not find container \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": container with ID starting with 3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568 not found: ID does not exist" Nov 02 08:07:15.846491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.846438 6160 scope.go:122] "RemoveContainer" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" Nov 02 08:07:15.846705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.846688 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2"} err="failed to get container status \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": rpc error: code = NotFound desc = could not find container \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": container with ID starting with 1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2 not found: ID does not exist" Nov 02 08:07:15.846749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.846705 6160 scope.go:122] "RemoveContainer" containerID="7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" Nov 02 08:07:15.846903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.846885 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607"} err="failed to get container status \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": rpc error: code = NotFound desc = could not find container \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": container with ID starting with 7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607 not found: ID does not exist" Nov 02 08:07:15.846943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.846904 6160 scope.go:122] "RemoveContainer" containerID="be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" Nov 02 08:07:15.847149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.847134 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224"} err="failed to get container status \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": rpc error: code = NotFound desc = could not find container \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": container with ID starting with be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224 not found: ID does not exist" Nov 02 08:07:15.847149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.847149 6160 scope.go:122] "RemoveContainer" containerID="f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" Nov 02 08:07:15.847415 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.847399 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5"} err="failed to get container status \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": rpc error: code = NotFound desc = could not find container \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": container with ID starting with f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5 not found: ID does not exist" Nov 02 08:07:15.847460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.847415 6160 scope.go:122] "RemoveContainer" containerID="d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" Nov 02 08:07:15.847628 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.847609 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e"} err="failed to get container status \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": rpc error: code = NotFound desc = could not find container \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": container with ID starting with d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e not found: ID does not exist" Nov 02 08:07:15.847677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.847629 6160 scope.go:122] "RemoveContainer" containerID="e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" Nov 02 08:07:15.847831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.847811 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7"} err="failed to get container status \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": rpc error: code = NotFound desc = could not find container \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": container with ID starting with e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7 not found: ID does not exist" Nov 02 08:07:15.847877 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.847831 6160 scope.go:122] "RemoveContainer" containerID="3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" Nov 02 08:07:15.848018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.847999 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568"} err="failed to get container status \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": rpc error: code = NotFound desc = could not find container \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": container with ID starting with 3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568 not found: ID does not exist" Nov 02 08:07:15.848053 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.848019 6160 scope.go:122] "RemoveContainer" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" Nov 02 08:07:15.848243 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.848228 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2"} err="failed to get container status \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": rpc error: code = NotFound desc = could not find container \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": container with ID starting with 1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2 not found: ID does not exist" Nov 02 08:07:15.848243 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.848242 6160 scope.go:122] "RemoveContainer" containerID="7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607" Nov 02 08:07:15.848475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.848455 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607"} err="failed to get container status \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": rpc error: code = NotFound desc = could not find container \"7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607\": container with ID starting with 7bcdbc767f3e47602043276de3e873b68c473c1868521e58365aee75b2bb3607 not found: ID does not exist" Nov 02 08:07:15.848546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.848487 6160 scope.go:122] "RemoveContainer" containerID="be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224" Nov 02 08:07:15.848713 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.848693 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224"} err="failed to get container status \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": rpc error: code = NotFound desc = could not find container \"be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224\": container with ID starting with be0802f36e33bc0985f522a1a6617377e8b5e78715b6976d974a45a78ce10224 not found: ID does not exist" Nov 02 08:07:15.848713 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.848712 6160 scope.go:122] "RemoveContainer" containerID="f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5" Nov 02 08:07:15.848899 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.848883 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5"} err="failed to get container status \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": rpc error: code = NotFound desc = could not find container \"f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5\": container with ID starting with f23e35694b1e4931cbaed48a6013b2c7af499c89ee90fe8b08f77969354561a5 not found: ID does not exist" Nov 02 08:07:15.848935 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.848900 6160 scope.go:122] "RemoveContainer" containerID="d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e" Nov 02 08:07:15.849110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.849091 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e"} err="failed to get container status \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": rpc error: code = NotFound desc = could not find container \"d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e\": container with ID starting with d861ff5184d935897c7b67e4a7f33184d02be8eabcb376ffcf230790ef6cff1e not found: ID does not exist" Nov 02 08:07:15.849192 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.849112 6160 scope.go:122] "RemoveContainer" containerID="e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7" Nov 02 08:07:15.849351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.849335 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7"} err="failed to get container status \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": rpc error: code = NotFound desc = could not find container \"e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7\": container with ID starting with e0c10aa391f917b2581effab1628a2e23ca902f54cffd415441a89a3f6295bf7 not found: ID does not exist" Nov 02 08:07:15.849399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.849351 6160 scope.go:122] "RemoveContainer" containerID="3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568" Nov 02 08:07:15.849547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.849530 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568"} err="failed to get container status \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": rpc error: code = NotFound desc = could not find container \"3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568\": container with ID starting with 3b513affcb8b76515aa37ea1fba0ccb7d1c3ece088832d761775961b198f6568 not found: ID does not exist" Nov 02 08:07:15.849596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.849548 6160 scope.go:122] "RemoveContainer" containerID="1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2" Nov 02 08:07:15.849747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.849728 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2"} err="failed to get container status \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": rpc error: code = NotFound desc = could not find container \"1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2\": container with ID starting with 1d7a9709fd3a16424dc793380fb64fb174afa4a4e74923bdc4ffd139a040ffa2 not found: ID does not exist" Nov 02 08:07:15.946501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.946466 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 08:07:15.993611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.993546 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 08:07:15.994386 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994357 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 08:07:15.994587 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994576 6160 cpu_manager.go:414] "containerMap: removing container" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="thanos-sidecar" Nov 02 08:07:15.994626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994589 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="thanos-sidecar" Nov 02 08:07:15.994626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994601 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="prom-label-proxy" Nov 02 08:07:15.994626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994606 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="prom-label-proxy" Nov 02 08:07:15.994626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994614 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy-metric" Nov 02 08:07:15.994626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994620 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy-metric" Nov 02 08:07:15.994626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994627 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="config-reloader" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994631 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="config-reloader" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994639 6160 cpu_manager.go:414] "containerMap: removing container" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="init-config-reloader" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994643 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="init-config-reloader" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994650 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="init-config-reloader" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994654 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="init-config-reloader" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994661 6160 cpu_manager.go:414] "containerMap: removing container" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="prometheus" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994665 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="prometheus" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994670 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="alertmanager" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994674 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="alertmanager" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994681 6160 cpu_manager.go:414] "containerMap: removing container" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="config-reloader" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994685 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="config-reloader" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994690 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy-web" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994694 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy-web" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994703 6160 cpu_manager.go:414] "containerMap: removing container" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy-web" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994707 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy-web" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994714 6160 cpu_manager.go:414] "containerMap: removing container" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994718 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994723 6160 cpu_manager.go:414] "containerMap: removing container" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy-thanos" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994727 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy-thanos" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994739 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994743 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994784 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="config-reloader" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994790 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy-web" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994796 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy-web" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994801 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994805 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="kube-rbac-proxy-thanos" Nov 02 08:07:15.994800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994812 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy" Nov 02 08:07:15.995555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994817 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="thanos-sidecar" Nov 02 08:07:15.995555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994823 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="prom-label-proxy" Nov 02 08:07:15.995555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994828 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="kube-rbac-proxy-metric" Nov 02 08:07:15.995555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994832 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="config-reloader" Nov 02 08:07:15.995555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994837 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="init-config-reloader" Nov 02 08:07:15.995555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994841 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" containerName="prometheus" Nov 02 08:07:15.995555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.994845 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a387bc4-fe09-4921-9ff4-8199642acb38" containerName="alertmanager" Nov 02 08:07:15.998274 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:15.998257 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.046988 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.046922 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 08:07:16.047329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.047166 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 08:07:16.047329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.047262 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 08:07:16.047964 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.047949 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 08:07:16.048123 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048108 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 08:07:16.048163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048112 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 08:07:16.048364 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048349 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 08:07:16.048498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048483 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xbrch\"" Nov 02 08:07:16.048557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048528 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 08:07:16.048557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048530 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 08:07:16.048650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048543 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 08:07:16.048650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048573 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmnro8141ghht\"" Nov 02 08:07:16.048650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048594 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 08:07:16.048878 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048865 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 08:07:16.048964 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.048949 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 08:07:16.095696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.095650 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 08:07:16.095696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.095651 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 08:07:16.096437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.096418 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 08:07:16.099408 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099385 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099424 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099458 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099507 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099621 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099642 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099665 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13f5952e-7c2f-41b1-919e-725db0078f60-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099680 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-web-config\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099763 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099736 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099777 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099800 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099832 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099869 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099886 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.099993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099947 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13f5952e-7c2f-41b1-919e-725db0078f60-config-out\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.100045 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.099990 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.100095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.100050 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.100149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.100089 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.100149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.100122 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcdr\" (UniqueName: \"kubernetes.io/projected/13f5952e-7c2f-41b1-919e-725db0078f60-kube-api-access-tvcdr\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.100243 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.100156 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-config\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200593 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200563 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200593 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200595 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200615 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200660 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200693 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13f5952e-7c2f-41b1-919e-725db0078f60-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200716 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-web-config\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200738 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200773 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200792 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200811 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.200837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200839 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.201613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200860 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.201613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200887 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13f5952e-7c2f-41b1-919e-725db0078f60-config-out\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.201613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200917 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.201613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200943 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.202102 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.202139 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.200972 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.202270 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcdr\" (UniqueName: \"kubernetes.io/projected/13f5952e-7c2f-41b1-919e-725db0078f60-kube-api-access-tvcdr\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.202313 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-config\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.202349 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.202359 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.202380 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.202989 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.203411 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.206147 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.206538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.206546 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.207371 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.206842 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.207430 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.207392 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13f5952e-7c2f-41b1-919e-725db0078f60-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.208842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.207696 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.208842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.208398 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.208842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.208729 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-web-config\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.209053 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.208853 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13f5952e-7c2f-41b1-919e-725db0078f60-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.209053 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.209013 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-config\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.209167 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.209050 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13f5952e-7c2f-41b1-919e-725db0078f60-config-out\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.209326 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.209297 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.210001 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.209971 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.210154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.210138 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.210403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.210387 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13f5952e-7c2f-41b1-919e-725db0078f60-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.255596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.255564 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcdr\" (UniqueName: \"kubernetes.io/projected/13f5952e-7c2f-41b1-919e-725db0078f60-kube-api-access-tvcdr\") pod \"prometheus-k8s-0\" (UID: \"13f5952e-7c2f-41b1-919e-725db0078f60\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.313730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.313654 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:16.728158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.728124 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 08:07:16.732376 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:07:16.732344 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f5952e_7c2f_41b1_919e_725db0078f60.slice/crio-6073f82f3839573219c21a70fb0df7ba3a628bcb40b5d0f6c1ef9a208d0c5d8d WatchSource:0}: Error finding container 6073f82f3839573219c21a70fb0df7ba3a628bcb40b5d0f6c1ef9a208d0c5d8d: Status 404 returned error can't find the container with id 6073f82f3839573219c21a70fb0df7ba3a628bcb40b5d0f6c1ef9a208d0c5d8d Nov 02 08:07:16.795921 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.795896 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13f5952e-7c2f-41b1-919e-725db0078f60","Type":"ContainerStarted","Data":"c503704dd631caf93f7a9dee05e7139f5944501ad3bdf58587dd4f1acc0fac59"} Nov 02 08:07:16.796061 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.795954 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13f5952e-7c2f-41b1-919e-725db0078f60","Type":"ContainerStarted","Data":"6073f82f3839573219c21a70fb0df7ba3a628bcb40b5d0f6c1ef9a208d0c5d8d"} Nov 02 08:07:16.918907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:16.918881 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:17.129156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:17.129069 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:17.580471 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:17.580431 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a4044b-18d1-494c-b50e-2b3d253efdf2" path="/var/lib/kubelet/pods/40a4044b-18d1-494c-b50e-2b3d253efdf2/volumes" Nov 02 08:07:17.801651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:17.801619 6160 generic.go:356] "Generic (PLEG): container finished" podID="13f5952e-7c2f-41b1-919e-725db0078f60" containerID="c503704dd631caf93f7a9dee05e7139f5944501ad3bdf58587dd4f1acc0fac59" exitCode=0 Nov 02 08:07:17.801859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:17.801715 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13f5952e-7c2f-41b1-919e-725db0078f60","Type":"ContainerDied","Data":"c503704dd631caf93f7a9dee05e7139f5944501ad3bdf58587dd4f1acc0fac59"} Nov 02 08:07:18.812921 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:18.812883 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13f5952e-7c2f-41b1-919e-725db0078f60","Type":"ContainerStarted","Data":"f6285cfbe10722aefed9424f79c38148d9c8089807bf500864b7ca8cc157b97c"} Nov 02 08:07:18.813295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:18.812978 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13f5952e-7c2f-41b1-919e-725db0078f60","Type":"ContainerStarted","Data":"b960c9ed49ea267f73ab58c22930ffbaf07d685c68e7e085c5b0d644f5f24711"} Nov 02 08:07:18.813295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:18.813004 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13f5952e-7c2f-41b1-919e-725db0078f60","Type":"ContainerStarted","Data":"44c2ba39630017e74def8d63713f15ae05c7f16a52a092b92d95c0c189922450"} Nov 02 08:07:18.813295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:18.813028 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13f5952e-7c2f-41b1-919e-725db0078f60","Type":"ContainerStarted","Data":"e7aae5ad92b062a0adbcdbbbb91a6b44b9d7b98d012741413ee984686d5b42be"} Nov 02 08:07:18.813295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:18.813050 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13f5952e-7c2f-41b1-919e-725db0078f60","Type":"ContainerStarted","Data":"7b9a9f177f52ca52e871e26a5c84c710333e752b2fab0c0e6d4ad53447343902"} Nov 02 08:07:18.813295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:18.813072 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13f5952e-7c2f-41b1-919e-725db0078f60","Type":"ContainerStarted","Data":"5e07b9142dcdf95cef84ec4fdc5351d9eacc26060f128e3b45c7ef07c9756bb3"} Nov 02 08:07:18.930299 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:18.930251 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.9129447710000003 podStartE2EDuration="3.93023622s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.291449ms" isStatefulPod=false podCreationTimestamp="2026-11-02 08:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:07:18.928604348 +0000 UTC m=+37.482108598" watchObservedRunningTime="2026-11-02 08:07:18.93023622 +0000 UTC m=+37.483740418" Nov 02 08:07:19.732699 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:19.732640 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:19.732923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:19.732901 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:19.737420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:19.737400 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:19.820316 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:19.820285 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fd5dcfd5d-zhfd5" Nov 02 08:07:19.988535 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:19.988455 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dd89b74f-67l4d"] Nov 02 08:07:20.757302 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:20.757269 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:20.913604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:20.913565 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:21.314350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:21.314301 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:40.520001 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:40.519965 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:07:42.783030 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:42.782994 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:42.936714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:42.936673 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:43.674047 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:43.674007 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:45.008436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.008370 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5dd89b74f-67l4d" podUID="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" containerName="console" containerID="cri-o://97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071" gracePeriod=15 Nov 02 08:07:45.137719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.137680 6160 patch_prober.go:29] interesting pod/console-5dd89b74f-67l4d container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.13:8443/health\": dial tcp 10.133.0.13:8443: connect: connection refused" start-of-body= Nov 02 08:07:45.137875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.137746 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5dd89b74f-67l4d" podUID="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" containerName="console" probeResult="failure" output="Get \"https://10.133.0.13:8443/health\": dial tcp 10.133.0.13:8443: connect: connection refused" Nov 02 08:07:45.261198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.261125 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:07:45.437847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.437809 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr\" (UniqueName: \"kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr\") pod \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " Nov 02 08:07:45.437847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.437850 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config\") pod \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " Nov 02 08:07:45.438097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.437871 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert\") pod \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " Nov 02 08:07:45.438097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.437891 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config\") pod \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " Nov 02 08:07:45.438097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.437923 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert\") pod \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " Nov 02 08:07:45.438097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.437979 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca\") pod \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " Nov 02 08:07:45.438097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.438027 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle\") pod \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " Nov 02 08:07:45.438097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.438071 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp\") pod \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\" (UID: \"3d6a0a75-e046-4589-8711-4a0f1fd83a9d\") " Nov 02 08:07:45.438489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.438460 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config" pod "3d6a0a75-e046-4589-8711-4a0f1fd83a9d" (UID: "3d6a0a75-e046-4589-8711-4a0f1fd83a9d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:45.438548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.438520 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp" pod "3d6a0a75-e046-4589-8711-4a0f1fd83a9d" (UID: "3d6a0a75-e046-4589-8711-4a0f1fd83a9d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:07:45.438602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.438582 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle" pod "3d6a0a75-e046-4589-8711-4a0f1fd83a9d" (UID: "3d6a0a75-e046-4589-8711-4a0f1fd83a9d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:45.438602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.438581 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca" pod "3d6a0a75-e046-4589-8711-4a0f1fd83a9d" (UID: "3d6a0a75-e046-4589-8711-4a0f1fd83a9d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:45.438870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.438671 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert" pod "3d6a0a75-e046-4589-8711-4a0f1fd83a9d" (UID: "3d6a0a75-e046-4589-8711-4a0f1fd83a9d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:07:45.440438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.440414 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr" pod "3d6a0a75-e046-4589-8711-4a0f1fd83a9d" (UID: "3d6a0a75-e046-4589-8711-4a0f1fd83a9d"). InnerVolumeSpecName "kube-api-access-zzpcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:07:45.440438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.440418 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config" pod "3d6a0a75-e046-4589-8711-4a0f1fd83a9d" (UID: "3d6a0a75-e046-4589-8711-4a0f1fd83a9d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:45.440595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.440494 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert" pod "3d6a0a75-e046-4589-8711-4a0f1fd83a9d" (UID: "3d6a0a75-e046-4589-8711-4a0f1fd83a9d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:07:45.539262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.539148 6160 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:45.539262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.539211 6160 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-trusted-ca-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:45.539262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.539222 6160 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-tmp\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:45.539262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.539234 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzpcr\" (UniqueName: \"kubernetes.io/projected/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-kube-api-access-zzpcr\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:45.539262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.539244 6160 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-config\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:45.539262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.539252 6160 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-serving-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:45.539262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.539262 6160 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-console-oauth-config\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:45.539262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.539270 6160 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d6a0a75-e046-4589-8711-4a0f1fd83a9d-oauth-serving-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:07:45.901153 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.901068 6160 generic.go:356] "Generic (PLEG): container finished" podID="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" containerID="97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071" exitCode=0 Nov 02 08:07:45.901153 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.901114 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd89b74f-67l4d" Nov 02 08:07:45.901414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.901117 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd89b74f-67l4d" event={"ID":"3d6a0a75-e046-4589-8711-4a0f1fd83a9d","Type":"ContainerDied","Data":"97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071"} Nov 02 08:07:45.901414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.901250 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd89b74f-67l4d" event={"ID":"3d6a0a75-e046-4589-8711-4a0f1fd83a9d","Type":"ContainerDied","Data":"70f2921a186f179e92aa7d509524b1302dbf6d0b7e3f26cba55a9e2b9644c307"} Nov 02 08:07:45.901414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.901286 6160 scope.go:122] "RemoveContainer" containerID="97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071" Nov 02 08:07:45.908950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.908929 6160 scope.go:122] "RemoveContainer" containerID="97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071" Nov 02 08:07:45.909251 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:07:45.909215 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071\": container with ID starting with 97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071 not found: ID does not exist" containerID="97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071" Nov 02 08:07:45.909327 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:45.909263 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071"} err="failed to get container status \"97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071\": rpc error: code = NotFound desc = could not find container \"97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071\": container with ID starting with 97d98114ee09132fba63a13d9cf98e8aac600f6f56943199d6fc097e4b6c8071 not found: ID does not exist" Nov 02 08:07:46.051834 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:46.051744 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dd89b74f-67l4d"] Nov 02 08:07:46.097566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:46.097536 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dd89b74f-67l4d"] Nov 02 08:07:46.254873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:46.254830 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:46.271051 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:46.271021 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:46.329673 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:46.329639 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 08:07:47.580602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:47.580569 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" path="/var/lib/kubelet/pods/3d6a0a75-e046-4589-8711-4a0f1fd83a9d/volumes" Nov 02 08:07:48.124594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:48.124554 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:53.449678 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:53.449639 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:59.730331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:07:59.730293 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:10.519549 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:08:10.519510 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:08:40.519588 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:08:40.519553 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:08:58.495201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:08:58.495154 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:58.655947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:08:58.655902 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:10.518893 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:09:10.518853 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:09:39.469922 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:39.469888 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:39.722101 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:39.722014 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:40.518938 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:09:40.518904 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:09:40.847494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:40.847405 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:41.066693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.066656 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:41.490337 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.490289 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6"] Nov 02 08:09:41.490740 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.490717 6160 cpu_manager.go:414] "containerMap: removing container" podUID="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" containerName="console" Nov 02 08:09:41.490922 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.490898 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" containerName="console" Nov 02 08:09:41.491046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.491007 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6a0a75-e046-4589-8711-4a0f1fd83a9d" containerName="console" Nov 02 08:09:41.493911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.493896 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.542234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.542203 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-c6nz9\"" Nov 02 08:09:41.588428 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.588399 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 08:09:41.588622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.588431 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 08:09:41.588978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.588958 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6"] Nov 02 08:09:41.593444 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.593426 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88"] Nov 02 08:09:41.596775 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.596756 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.603615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.603596 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmgw\" (UniqueName: \"kubernetes.io/projected/c27216d9-90ea-4bc4-badd-c9a03ce61a57-kube-api-access-7kmgw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.603702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.603632 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.603762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.603746 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.646001 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.645978 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-7pj26\"" Nov 02 08:09:41.650772 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.650754 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88"] Nov 02 08:09:41.688866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.688812 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph"] Nov 02 08:09:41.692287 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.692267 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:41.704620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.704598 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nff9k\" (UniqueName: \"kubernetes.io/projected/27b82ea9-bab0-42fa-9d34-662bc2e89254-kube-api-access-nff9k\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.704732 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.704634 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.704732 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.704718 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.704847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.704773 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.704847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.704798 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmgw\" (UniqueName: \"kubernetes.io/projected/c27216d9-90ea-4bc4-badd-c9a03ce61a57-kube-api-access-7kmgw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.704847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.704819 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.705048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.705029 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.705259 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.705089 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.740299 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.740273 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-7smwn\"" Nov 02 08:09:41.747127 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.747077 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph"] Nov 02 08:09:41.758229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.758208 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmgw\" (UniqueName: \"kubernetes.io/projected/c27216d9-90ea-4bc4-badd-c9a03ce61a57-kube-api-access-7kmgw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.791917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.791874 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h"] Nov 02 08:09:41.795290 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.795268 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:41.805377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.805353 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nff9k\" (UniqueName: \"kubernetes.io/projected/27b82ea9-bab0-42fa-9d34-662bc2e89254-kube-api-access-nff9k\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.805478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.805407 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.805478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.805443 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.805554 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.805483 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:41.805554 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.805510 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:41.805554 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.805539 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgnqt\" (UniqueName: \"kubernetes.io/projected/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-kube-api-access-lgnqt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:41.805943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.805923 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.805979 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.805936 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.808354 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.808337 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:09:41.809442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.809419 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:41.844106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.844015 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-4s7bt\"" Nov 02 08:09:41.849590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.849561 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h"] Nov 02 08:09:41.859390 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.858482 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nff9k\" (UniqueName: \"kubernetes.io/projected/27b82ea9-bab0-42fa-9d34-662bc2e89254-kube-api-access-nff9k\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.906558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.906525 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:41.906558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.906557 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:41.906779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.906578 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnqt\" (UniqueName: \"kubernetes.io/projected/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-kube-api-access-lgnqt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:41.906779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.906609 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:41.906779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.906646 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:41.906779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.906679 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrdq\" (UniqueName: \"kubernetes.io/projected/92e17c7b-6cb5-4848-a75e-4e633a2297f0-kube-api-access-jgrdq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:41.906985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.906966 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:41.907017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.906992 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:41.912688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.912663 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:09:41.961701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:41.961061 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnqt\" (UniqueName: \"kubernetes.io/projected/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-kube-api-access-lgnqt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:42.007730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.007630 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:09:42.007904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.007809 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:42.007904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.007854 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:42.007904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.007884 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrdq\" (UniqueName: \"kubernetes.io/projected/92e17c7b-6cb5-4848-a75e-4e633a2297f0-kube-api-access-jgrdq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:42.008194 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.008156 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:42.008242 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.008216 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:42.062963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.062323 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrdq\" (UniqueName: \"kubernetes.io/projected/92e17c7b-6cb5-4848-a75e-4e633a2297f0-kube-api-access-jgrdq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:42.112032 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.112001 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:09:42.174233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.174207 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6"] Nov 02 08:09:42.180636 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:42.180610 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27216d9_90ea_4bc4_badd_c9a03ce61a57.slice/crio-7db6625827aa328a5f17e27d35f88da500e85872a81e5c7d1cd4a038fcb9ffda WatchSource:0}: Error finding container 7db6625827aa328a5f17e27d35f88da500e85872a81e5c7d1cd4a038fcb9ffda: Status 404 returned error can't find the container with id 7db6625827aa328a5f17e27d35f88da500e85872a81e5c7d1cd4a038fcb9ffda Nov 02 08:09:42.270874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.270787 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" event={"ID":"c27216d9-90ea-4bc4-badd-c9a03ce61a57","Type":"ContainerStarted","Data":"7db6625827aa328a5f17e27d35f88da500e85872a81e5c7d1cd4a038fcb9ffda"} Nov 02 08:09:42.275536 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.275511 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88"] Nov 02 08:09:42.279143 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:42.279120 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b82ea9_bab0_42fa_9d34_662bc2e89254.slice/crio-bd97e10394b4cbed7fbbc9924bcd1a3f9e7a73009e48e8e5e3fb5628e37e7574 WatchSource:0}: Error finding container bd97e10394b4cbed7fbbc9924bcd1a3f9e7a73009e48e8e5e3fb5628e37e7574: Status 404 returned error can't find the container with id bd97e10394b4cbed7fbbc9924bcd1a3f9e7a73009e48e8e5e3fb5628e37e7574 Nov 02 08:09:42.373749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.373666 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph"] Nov 02 08:09:42.377355 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:42.377322 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7cc0a4_6464_4d62_b076_e6dec51b41d3.slice/crio-cefee873a2f861248273735a789018ae8bd0714f2b0abd6289b095f88c02fdba WatchSource:0}: Error finding container cefee873a2f861248273735a789018ae8bd0714f2b0abd6289b095f88c02fdba: Status 404 returned error can't find the container with id cefee873a2f861248273735a789018ae8bd0714f2b0abd6289b095f88c02fdba Nov 02 08:09:42.484257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:42.484224 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h"] Nov 02 08:09:42.487593 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:42.487567 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e17c7b_6cb5_4848_a75e_4e633a2297f0.slice/crio-aa36e6e46dbc1802b9d34b5a6ef72aad8d71abb3fb338d48bc1461a0528f6477 WatchSource:0}: Error finding container aa36e6e46dbc1802b9d34b5a6ef72aad8d71abb3fb338d48bc1461a0528f6477: Status 404 returned error can't find the container with id aa36e6e46dbc1802b9d34b5a6ef72aad8d71abb3fb338d48bc1461a0528f6477 Nov 02 08:09:43.028735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.028697 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf"] Nov 02 08:09:43.033781 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.033758 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.082360 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.082323 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-m98s7\"" Nov 02 08:09:43.086866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.086835 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf"] Nov 02 08:09:43.117368 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.117334 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgnf2\" (UniqueName: \"kubernetes.io/projected/102f9b5d-cabe-4703-befb-33eeda727d34-kube-api-access-pgnf2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.117526 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.117401 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.117526 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.117510 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.218116 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.218078 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgnf2\" (UniqueName: \"kubernetes.io/projected/102f9b5d-cabe-4703-befb-33eeda727d34-kube-api-access-pgnf2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.218321 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.218139 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.218321 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.218161 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.218533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.218516 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.218618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.218596 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.271831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.271805 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgnf2\" (UniqueName: \"kubernetes.io/projected/102f9b5d-cabe-4703-befb-33eeda727d34-kube-api-access-pgnf2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.278257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.278221 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" event={"ID":"92e17c7b-6cb5-4848-a75e-4e633a2297f0","Type":"ContainerStarted","Data":"aa36e6e46dbc1802b9d34b5a6ef72aad8d71abb3fb338d48bc1461a0528f6477"} Nov 02 08:09:43.279730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.279658 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" event={"ID":"27b82ea9-bab0-42fa-9d34-662bc2e89254","Type":"ContainerStarted","Data":"bd97e10394b4cbed7fbbc9924bcd1a3f9e7a73009e48e8e5e3fb5628e37e7574"} Nov 02 08:09:43.281000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.280965 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" event={"ID":"2b7cc0a4-6464-4d62-b076-e6dec51b41d3","Type":"ContainerStarted","Data":"cefee873a2f861248273735a789018ae8bd0714f2b0abd6289b095f88c02fdba"} Nov 02 08:09:43.349113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.349061 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:09:43.719452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:43.719416 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf"] Nov 02 08:09:43.723787 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:43.723750 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102f9b5d_cabe_4703_befb_33eeda727d34.slice/crio-b2513fec4b7ccd13d50797324031e677084418db49022a7847815858a2997072 WatchSource:0}: Error finding container b2513fec4b7ccd13d50797324031e677084418db49022a7847815858a2997072: Status 404 returned error can't find the container with id b2513fec4b7ccd13d50797324031e677084418db49022a7847815858a2997072 Nov 02 08:09:44.288707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.288611 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" event={"ID":"102f9b5d-cabe-4703-befb-33eeda727d34","Type":"ContainerStarted","Data":"b2513fec4b7ccd13d50797324031e677084418db49022a7847815858a2997072"} Nov 02 08:09:44.617235 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.617132 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n"] Nov 02 08:09:44.631092 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.631057 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.675875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.675132 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n"] Nov 02 08:09:44.720441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.720414 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-2kfnz\"" Nov 02 08:09:44.732986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.732953 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5pqc\" (UniqueName: \"kubernetes.io/projected/c3617eb7-9969-4430-bf56-0da8db1e9e28-kube-api-access-m5pqc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.733158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.733016 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.733158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.733076 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.834902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.833796 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5pqc\" (UniqueName: \"kubernetes.io/projected/c3617eb7-9969-4430-bf56-0da8db1e9e28-kube-api-access-m5pqc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.834902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.833843 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.834902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.833883 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.834902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.834273 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.834902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.834851 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.893157 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.893085 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5pqc\" (UniqueName: \"kubernetes.io/projected/c3617eb7-9969-4430-bf56-0da8db1e9e28-kube-api-access-m5pqc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:44.952361 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:44.952317 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:09:45.355054 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:45.355017 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n"] Nov 02 08:09:45.369017 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:45.368983 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3617eb7_9969_4430_bf56_0da8db1e9e28.slice/crio-15d3bd5b0e391c067913c2b1ad3d5b96f17363a2569d0b0c526aad41132e893e WatchSource:0}: Error finding container 15d3bd5b0e391c067913c2b1ad3d5b96f17363a2569d0b0c526aad41132e893e: Status 404 returned error can't find the container with id 15d3bd5b0e391c067913c2b1ad3d5b96f17363a2569d0b0c526aad41132e893e Nov 02 08:09:46.301735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:46.301687 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" event={"ID":"c3617eb7-9969-4430-bf56-0da8db1e9e28","Type":"ContainerStarted","Data":"15d3bd5b0e391c067913c2b1ad3d5b96f17363a2569d0b0c526aad41132e893e"} Nov 02 08:09:47.328620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:47.328525 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:47.561820 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:47.561605 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:47.801930 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:47.801839 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:47.992654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:47.992596 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:48.160672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:48.160565 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:49.031475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:49.031409 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:51.865286 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:51.865061 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:52.217968 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.217933 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z"] Nov 02 08:09:52.223076 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.223049 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.272159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.271725 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ndh4j\"" Nov 02 08:09:52.316392 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.316353 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.316581 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.316414 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhsr\" (UniqueName: \"kubernetes.io/projected/99ea3be8-4201-426e-aadd-609d23386998-kube-api-access-mfhsr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.316581 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.316460 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.318985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.318959 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z"] Nov 02 08:09:52.417575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.417322 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.417575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.417399 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhsr\" (UniqueName: \"kubernetes.io/projected/99ea3be8-4201-426e-aadd-609d23386998-kube-api-access-mfhsr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.417575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.417467 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.417881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.417790 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.418114 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.418092 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.473433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.473311 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhsr\" (UniqueName: \"kubernetes.io/projected/99ea3be8-4201-426e-aadd-609d23386998-kube-api-access-mfhsr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.545883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.545838 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:09:52.948670 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:52.948563 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z"] Nov 02 08:09:52.951877 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:52.951844 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ea3be8_4201_426e_aadd_609d23386998.slice/crio-bb4cb9ebf03f085dc35cf5bc45c783b5e3ff9891e969367d4c3b1ca7afcd9dc8 WatchSource:0}: Error finding container bb4cb9ebf03f085dc35cf5bc45c783b5e3ff9891e969367d4c3b1ca7afcd9dc8: Status 404 returned error can't find the container with id bb4cb9ebf03f085dc35cf5bc45c783b5e3ff9891e969367d4c3b1ca7afcd9dc8 Nov 02 08:09:53.355368 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:53.355256 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" event={"ID":"99ea3be8-4201-426e-aadd-609d23386998","Type":"ContainerStarted","Data":"bb4cb9ebf03f085dc35cf5bc45c783b5e3ff9891e969367d4c3b1ca7afcd9dc8"} Nov 02 08:09:54.946740 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:54.946701 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc"] Nov 02 08:09:54.958428 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:54.958395 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:54.978340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:54.978306 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6"] Nov 02 08:09:54.991930 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:54.991904 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.001790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.001767 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr"] Nov 02 08:09:55.012655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.012632 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc"] Nov 02 08:09:55.012786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.012733 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.029203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.028365 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7"] Nov 02 08:09:55.040089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.039836 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.040089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.039993 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.040089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.040054 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.040089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.040088 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.040417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.040123 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8w5\" (UniqueName: \"kubernetes.io/projected/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-kube-api-access-dk8w5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.040417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.040169 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dwd\" (UniqueName: \"kubernetes.io/projected/2ee7da9e-0167-4577-bfc6-05921d3c0233-kube-api-access-89dwd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.040417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.040217 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbr6v\" (UniqueName: \"kubernetes.io/projected/1531694f-d551-47a5-93d8-d3efd6a87e36-kube-api-access-jbr6v\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.040417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.040264 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.040417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.040302 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.049226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.049195 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.062623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.062598 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr"] Nov 02 08:09:55.077206 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.077126 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6"] Nov 02 08:09:55.092530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.092455 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7"] Nov 02 08:09:55.140990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.140965 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.141107 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141010 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.141351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141318 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z754t\" (UniqueName: \"kubernetes.io/projected/cab83ba3-8d56-4572-badf-2c9f481faea6-kube-api-access-z754t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.141500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141386 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8w5\" (UniqueName: \"kubernetes.io/projected/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-kube-api-access-dk8w5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.141500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141400 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.141500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141487 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.141693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141535 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.141693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141544 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dwd\" (UniqueName: \"kubernetes.io/projected/2ee7da9e-0167-4577-bfc6-05921d3c0233-kube-api-access-89dwd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.141693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141575 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbr6v\" (UniqueName: \"kubernetes.io/projected/1531694f-d551-47a5-93d8-d3efd6a87e36-kube-api-access-jbr6v\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.141693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141611 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.141693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141648 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.141693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141682 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.142007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.141741 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.142073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.142055 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.142146 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.142075 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.142748 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.142354 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.142748 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.142392 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.142748 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.142692 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.195835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.195807 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dwd\" (UniqueName: \"kubernetes.io/projected/2ee7da9e-0167-4577-bfc6-05921d3c0233-kube-api-access-89dwd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.196618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.196597 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8w5\" (UniqueName: \"kubernetes.io/projected/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-kube-api-access-dk8w5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.196721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.196626 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbr6v\" (UniqueName: \"kubernetes.io/projected/1531694f-d551-47a5-93d8-d3efd6a87e36-kube-api-access-jbr6v\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.244870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.243268 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z754t\" (UniqueName: \"kubernetes.io/projected/cab83ba3-8d56-4572-badf-2c9f481faea6-kube-api-access-z754t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.244870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.243311 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.244870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.243350 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.244870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.243757 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.244870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.243884 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.297747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.297704 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z754t\" (UniqueName: \"kubernetes.io/projected/cab83ba3-8d56-4572-badf-2c9f481faea6-kube-api-access-z754t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.309720 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.309524 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:09:55.354430 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.354110 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:09:55.365210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.365160 6160 generic.go:356] "Generic (PLEG): container finished" podID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerID="8627fe8a30b864a6c49cc52a04a422a4ef8ea031742b79673da5a1ab908056df" exitCode=0 Nov 02 08:09:55.365350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.365253 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" event={"ID":"c27216d9-90ea-4bc4-badd-c9a03ce61a57","Type":"ContainerDied","Data":"8627fe8a30b864a6c49cc52a04a422a4ef8ea031742b79673da5a1ab908056df"} Nov 02 08:09:55.368639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.368613 6160 generic.go:356] "Generic (PLEG): container finished" podID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerID="541398ecc5685a4546670595fae71ccf557462ed06fa4fdcb8004a110a17beee" exitCode=0 Nov 02 08:09:55.368822 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.368726 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" event={"ID":"27b82ea9-bab0-42fa-9d34-662bc2e89254","Type":"ContainerDied","Data":"541398ecc5685a4546670595fae71ccf557462ed06fa4fdcb8004a110a17beee"} Nov 02 08:09:55.372701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.371973 6160 generic.go:356] "Generic (PLEG): container finished" podID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerID="d74657ce65aeb35de4ef3040b86692291b105579d0f39891e8786e818f633646" exitCode=0 Nov 02 08:09:55.372701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.372052 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" event={"ID":"2b7cc0a4-6464-4d62-b076-e6dec51b41d3","Type":"ContainerDied","Data":"d74657ce65aeb35de4ef3040b86692291b105579d0f39891e8786e818f633646"} Nov 02 08:09:55.404972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.403579 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:09:55.450969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.448343 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz"] Nov 02 08:09:55.468098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.467929 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:09:55.469903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.468738 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.517609 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.517528 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz"] Nov 02 08:09:55.546427 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.546383 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.546607 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.546471 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.546607 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.546508 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkm5d\" (UniqueName: \"kubernetes.io/projected/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-kube-api-access-gkm5d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.649851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.647335 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.649851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.647453 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.649851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.647485 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkm5d\" (UniqueName: \"kubernetes.io/projected/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-kube-api-access-gkm5d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.649851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.648692 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.649851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.649558 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.703962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.703879 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkm5d\" (UniqueName: \"kubernetes.io/projected/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-kube-api-access-gkm5d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.734215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.733710 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc"] Nov 02 08:09:55.740337 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:55.740306 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b21ebb0_f617_4854_a2db_cf113a0eb2bf.slice/crio-b8440c38406fadf4cf89f64d3e696272103a68b820295c98ba248d318b0e7991 WatchSource:0}: Error finding container b8440c38406fadf4cf89f64d3e696272103a68b820295c98ba248d318b0e7991: Status 404 returned error can't find the container with id b8440c38406fadf4cf89f64d3e696272103a68b820295c98ba248d318b0e7991 Nov 02 08:09:55.784818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.784791 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6"] Nov 02 08:09:55.833854 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.833829 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr"] Nov 02 08:09:55.836916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.836718 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:09:55.839697 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:55.839671 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee7da9e_0167_4577_bfc6_05921d3c0233.slice/crio-777c95ba72d21a28da5977c1e68682602841c2765b83798b1a44df1c5ef5fbdc WatchSource:0}: Error finding container 777c95ba72d21a28da5977c1e68682602841c2765b83798b1a44df1c5ef5fbdc: Status 404 returned error can't find the container with id 777c95ba72d21a28da5977c1e68682602841c2765b83798b1a44df1c5ef5fbdc Nov 02 08:09:55.885358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:55.885330 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7"] Nov 02 08:09:55.905213 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:55.904638 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab83ba3_8d56_4572_badf_2c9f481faea6.slice/crio-34d3a9a6b93ca66fc49cd623363c8622024c9674acc711ede876fca61adb6ac3 WatchSource:0}: Error finding container 34d3a9a6b93ca66fc49cd623363c8622024c9674acc711ede876fca61adb6ac3: Status 404 returned error can't find the container with id 34d3a9a6b93ca66fc49cd623363c8622024c9674acc711ede876fca61adb6ac3 Nov 02 08:09:56.264658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.264577 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz"] Nov 02 08:09:56.269730 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:09:56.269700 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99d10da_d327_4799_b1c2_ef2c23cb5aa1.slice/crio-edf3716c483c3878744ee7dccd48d9c79a4cb86389f9f8825046f54e1e588e29 WatchSource:0}: Error finding container edf3716c483c3878744ee7dccd48d9c79a4cb86389f9f8825046f54e1e588e29: Status 404 returned error can't find the container with id edf3716c483c3878744ee7dccd48d9c79a4cb86389f9f8825046f54e1e588e29 Nov 02 08:09:56.390603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.390518 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" event={"ID":"a99d10da-d327-4799-b1c2-ef2c23cb5aa1","Type":"ContainerStarted","Data":"edf3716c483c3878744ee7dccd48d9c79a4cb86389f9f8825046f54e1e588e29"} Nov 02 08:09:56.393287 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.392935 6160 generic.go:356] "Generic (PLEG): container finished" podID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerID="995d88e35a7d4091ca5d827428b403e170415a20aa062a9a72599eb13c12f153" exitCode=0 Nov 02 08:09:56.393287 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.393005 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" event={"ID":"cab83ba3-8d56-4572-badf-2c9f481faea6","Type":"ContainerDied","Data":"995d88e35a7d4091ca5d827428b403e170415a20aa062a9a72599eb13c12f153"} Nov 02 08:09:56.393287 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.393044 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" event={"ID":"cab83ba3-8d56-4572-badf-2c9f481faea6","Type":"ContainerStarted","Data":"34d3a9a6b93ca66fc49cd623363c8622024c9674acc711ede876fca61adb6ac3"} Nov 02 08:09:56.397452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.397002 6160 generic.go:356] "Generic (PLEG): container finished" podID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerID="e2ffb59fd362c816817de7de990d9ab614239548a2c6cfa8661a0017fe7c53f3" exitCode=0 Nov 02 08:09:56.397452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.397079 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" event={"ID":"1531694f-d551-47a5-93d8-d3efd6a87e36","Type":"ContainerDied","Data":"e2ffb59fd362c816817de7de990d9ab614239548a2c6cfa8661a0017fe7c53f3"} Nov 02 08:09:56.397452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.397121 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" event={"ID":"1531694f-d551-47a5-93d8-d3efd6a87e36","Type":"ContainerStarted","Data":"79884a5c0b2d8043fd732d49909600f59dbcdb9f5a0a9c17c49505fc0f67ff61"} Nov 02 08:09:56.400482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.399680 6160 generic.go:356] "Generic (PLEG): container finished" podID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerID="a41be03c2a9455567a8c9f8721a2126ea34496c6b34bf7abc6999e9bbf7353ec" exitCode=0 Nov 02 08:09:56.400482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.399755 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" event={"ID":"3b21ebb0-f617-4854-a2db-cf113a0eb2bf","Type":"ContainerDied","Data":"a41be03c2a9455567a8c9f8721a2126ea34496c6b34bf7abc6999e9bbf7353ec"} Nov 02 08:09:56.400482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.399791 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" event={"ID":"3b21ebb0-f617-4854-a2db-cf113a0eb2bf","Type":"ContainerStarted","Data":"b8440c38406fadf4cf89f64d3e696272103a68b820295c98ba248d318b0e7991"} Nov 02 08:09:56.404110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.404084 6160 generic.go:356] "Generic (PLEG): container finished" podID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerID="3f32b2a127b0341b2c9c0e9d1ab46e25dd8b485e10c8c656823f5daa8808e79d" exitCode=0 Nov 02 08:09:56.404222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.404187 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" event={"ID":"92e17c7b-6cb5-4848-a75e-4e633a2297f0","Type":"ContainerDied","Data":"3f32b2a127b0341b2c9c0e9d1ab46e25dd8b485e10c8c656823f5daa8808e79d"} Nov 02 08:09:56.408989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.408323 6160 generic.go:356] "Generic (PLEG): container finished" podID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerID="d00df5258574fb8644495b7b39617addd5f34e23718ceea014df756b66cc2bc5" exitCode=0 Nov 02 08:09:56.408989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.408376 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" event={"ID":"2ee7da9e-0167-4577-bfc6-05921d3c0233","Type":"ContainerDied","Data":"d00df5258574fb8644495b7b39617addd5f34e23718ceea014df756b66cc2bc5"} Nov 02 08:09:56.408989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:56.408413 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" event={"ID":"2ee7da9e-0167-4577-bfc6-05921d3c0233","Type":"ContainerStarted","Data":"777c95ba72d21a28da5977c1e68682602841c2765b83798b1a44df1c5ef5fbdc"} Nov 02 08:09:57.423474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:57.423437 6160 generic.go:356] "Generic (PLEG): container finished" podID="102f9b5d-cabe-4703-befb-33eeda727d34" containerID="6e5e8ab30ea90045da9d8a4546cb449b23b9ea2fd139376687aa451f5ad0501a" exitCode=0 Nov 02 08:09:57.423922 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:57.423583 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" event={"ID":"102f9b5d-cabe-4703-befb-33eeda727d34","Type":"ContainerDied","Data":"6e5e8ab30ea90045da9d8a4546cb449b23b9ea2fd139376687aa451f5ad0501a"} Nov 02 08:09:57.427742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:57.427101 6160 generic.go:356] "Generic (PLEG): container finished" podID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerID="999cb445ab278f56f13cfaada24b9761476985ffbd829584ca0e80ae3c037d18" exitCode=0 Nov 02 08:09:57.427742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:57.427194 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" event={"ID":"a99d10da-d327-4799-b1c2-ef2c23cb5aa1","Type":"ContainerDied","Data":"999cb445ab278f56f13cfaada24b9761476985ffbd829584ca0e80ae3c037d18"} Nov 02 08:09:58.433953 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.433922 6160 generic.go:356] "Generic (PLEG): container finished" podID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerID="2bfea2bfd8eaf96a0ff83d38b1c8ee9ccae626f38664f1723e8c9d9391721c50" exitCode=0 Nov 02 08:09:58.434432 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.434023 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" event={"ID":"3b21ebb0-f617-4854-a2db-cf113a0eb2bf","Type":"ContainerDied","Data":"2bfea2bfd8eaf96a0ff83d38b1c8ee9ccae626f38664f1723e8c9d9391721c50"} Nov 02 08:09:58.436212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.436098 6160 generic.go:356] "Generic (PLEG): container finished" podID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerID="00ee71bb4e87629d944120c0197d7873eea4ef05922c67512006c1a9d6b56814" exitCode=0 Nov 02 08:09:58.436212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.436195 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" event={"ID":"92e17c7b-6cb5-4848-a75e-4e633a2297f0","Type":"ContainerDied","Data":"00ee71bb4e87629d944120c0197d7873eea4ef05922c67512006c1a9d6b56814"} Nov 02 08:09:58.438598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.438576 6160 generic.go:356] "Generic (PLEG): container finished" podID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerID="340f1f89969d2e0c7de02b3f3a93d80c2b406a54fd8e749e6efede690a670b87" exitCode=0 Nov 02 08:09:58.438693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.438669 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" event={"ID":"c27216d9-90ea-4bc4-badd-c9a03ce61a57","Type":"ContainerDied","Data":"340f1f89969d2e0c7de02b3f3a93d80c2b406a54fd8e749e6efede690a670b87"} Nov 02 08:09:58.440905 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.440882 6160 generic.go:356] "Generic (PLEG): container finished" podID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerID="6fed54321065b041ddfee4dcb70f5edf5945dd27d8e3d0975f11eb8e9572d41b" exitCode=0 Nov 02 08:09:58.441006 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.440964 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" event={"ID":"27b82ea9-bab0-42fa-9d34-662bc2e89254","Type":"ContainerDied","Data":"6fed54321065b041ddfee4dcb70f5edf5945dd27d8e3d0975f11eb8e9572d41b"} Nov 02 08:09:58.442788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.442721 6160 generic.go:356] "Generic (PLEG): container finished" podID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerID="81e35198b8e7552744a6ffdc40f31b0f3e3cece2b4499cfbb138aa37da13ef57" exitCode=0 Nov 02 08:09:58.442856 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.442782 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" event={"ID":"c3617eb7-9969-4430-bf56-0da8db1e9e28","Type":"ContainerDied","Data":"81e35198b8e7552744a6ffdc40f31b0f3e3cece2b4499cfbb138aa37da13ef57"} Nov 02 08:09:58.445530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.445504 6160 generic.go:356] "Generic (PLEG): container finished" podID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerID="d2613dd25157dc665565e637dc80ba468e0e7aa8814aec2c5272fc52c7186396" exitCode=0 Nov 02 08:09:58.445818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.445796 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" event={"ID":"2ee7da9e-0167-4577-bfc6-05921d3c0233","Type":"ContainerDied","Data":"d2613dd25157dc665565e637dc80ba468e0e7aa8814aec2c5272fc52c7186396"} Nov 02 08:09:58.447913 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.447866 6160 generic.go:356] "Generic (PLEG): container finished" podID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerID="ba966177db9f2017d60c32689444674df893da4c7b318cf2e9f1642873225a8a" exitCode=0 Nov 02 08:09:58.447988 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.447939 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" event={"ID":"2b7cc0a4-6464-4d62-b076-e6dec51b41d3","Type":"ContainerDied","Data":"ba966177db9f2017d60c32689444674df893da4c7b318cf2e9f1642873225a8a"} Nov 02 08:09:58.450219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.450137 6160 generic.go:356] "Generic (PLEG): container finished" podID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerID="377a9052f1129fcf7965026e5ff3ce38486237acc10996232a5bf7e64de043ec" exitCode=0 Nov 02 08:09:58.450219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.450208 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" event={"ID":"cab83ba3-8d56-4572-badf-2c9f481faea6","Type":"ContainerDied","Data":"377a9052f1129fcf7965026e5ff3ce38486237acc10996232a5bf7e64de043ec"} Nov 02 08:09:58.452391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.452327 6160 generic.go:356] "Generic (PLEG): container finished" podID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerID="a7a891ad1f8944c7d3c0f062d760fda4391f42bf69b3cd16208e561875ab7895" exitCode=0 Nov 02 08:09:58.452391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.452351 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" event={"ID":"1531694f-d551-47a5-93d8-d3efd6a87e36","Type":"ContainerDied","Data":"a7a891ad1f8944c7d3c0f062d760fda4391f42bf69b3cd16208e561875ab7895"} Nov 02 08:09:58.803077 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:58.803050 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:59.461556 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:59.461510 6160 generic.go:356] "Generic (PLEG): container finished" podID="99ea3be8-4201-426e-aadd-609d23386998" containerID="e550575c13d1a4b08c2cf4da59571933fecfc2bd64cf1b2026ac6fc4597ae8c1" exitCode=0 Nov 02 08:09:59.462103 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:09:59.461634 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" event={"ID":"99ea3be8-4201-426e-aadd-609d23386998","Type":"ContainerDied","Data":"e550575c13d1a4b08c2cf4da59571933fecfc2bd64cf1b2026ac6fc4597ae8c1"} Nov 02 08:10:01.508023 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:01.506838 6160 generic.go:356] "Generic (PLEG): container finished" podID="99ea3be8-4201-426e-aadd-609d23386998" containerID="b5439258562629dada849b2ebdcd8af24b6d120294ed1ae5c0f4f62cfa744df9" exitCode=0 Nov 02 08:10:01.508023 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:01.506968 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" event={"ID":"99ea3be8-4201-426e-aadd-609d23386998","Type":"ContainerDied","Data":"b5439258562629dada849b2ebdcd8af24b6d120294ed1ae5c0f4f62cfa744df9"} Nov 02 08:10:07.624285 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:07.624242 6160 generic.go:356] "Generic (PLEG): container finished" podID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerID="c21d192ffbf9f2f185e152ec8abedb286f9f05ef5a57d030f41a3dab245e093b" exitCode=0 Nov 02 08:10:07.624753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:07.624396 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" event={"ID":"a99d10da-d327-4799-b1c2-ef2c23cb5aa1","Type":"ContainerDied","Data":"c21d192ffbf9f2f185e152ec8abedb286f9f05ef5a57d030f41a3dab245e093b"} Nov 02 08:10:07.643108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:07.643053 6160 generic.go:356] "Generic (PLEG): container finished" podID="102f9b5d-cabe-4703-befb-33eeda727d34" containerID="4375fa455976068dde4c3060118158f459bbbc53649624dc43b42b8ce4c2061b" exitCode=0 Nov 02 08:10:07.643234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:07.643143 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" event={"ID":"102f9b5d-cabe-4703-befb-33eeda727d34","Type":"ContainerDied","Data":"4375fa455976068dde4c3060118158f459bbbc53649624dc43b42b8ce4c2061b"} Nov 02 08:10:08.666277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:08.666238 6160 generic.go:356] "Generic (PLEG): container finished" podID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerID="1944a0685a2e7cf9bdaa5e92e75db1340e030ce0521c62ce70eb6fa53fe32770" exitCode=0 Nov 02 08:10:08.666761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:08.666401 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" event={"ID":"c3617eb7-9969-4430-bf56-0da8db1e9e28","Type":"ContainerDied","Data":"1944a0685a2e7cf9bdaa5e92e75db1340e030ce0521c62ce70eb6fa53fe32770"} Nov 02 08:10:10.554469 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:10:10.554423 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:10:20.343126 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:20.342766 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:22.049822 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:22.049787 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:23.876488 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:23.876404 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" event={"ID":"c27216d9-90ea-4bc4-badd-c9a03ce61a57","Type":"ContainerStarted","Data":"c2cf332d06a9030f4ea29f7e9e929282a263dc652f8b8d2955bb55e7cfa9f316"} Nov 02 08:10:24.007106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.007041 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" podStartSLOduration=2.677398959 podStartE2EDuration="43.007019611s" totalImagesPullingTime="40.329620652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:24.004383964 +0000 UTC m=+222.557888199" watchObservedRunningTime="2026-11-02 08:10:24.007019611 +0000 UTC m=+222.560523809" Nov 02 08:10:24.309073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.308347 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:24.627609 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.627396 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:24.885167 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.885066 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:10:24.885916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.885885 6160 generic.go:356] "Generic (PLEG): container finished" podID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerID="ef291ac2d048e68091868510ddc95f4a3f4f49f96a1fbb74279af5fdb0d8361d" exitCode=1 Nov 02 08:10:24.886039 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.885998 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" event={"ID":"a99d10da-d327-4799-b1c2-ef2c23cb5aa1","Type":"ContainerDied","Data":"ef291ac2d048e68091868510ddc95f4a3f4f49f96a1fbb74279af5fdb0d8361d"} Nov 02 08:10:24.889209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.889046 6160 generic.go:356] "Generic (PLEG): container finished" podID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerID="72e00f2cdae01dfa7e61c9d3070c7dd282964f9f2bd6788a46a2f8f97de4209f" exitCode=0 Nov 02 08:10:24.889209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.889134 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" event={"ID":"cab83ba3-8d56-4572-badf-2c9f481faea6","Type":"ContainerDied","Data":"72e00f2cdae01dfa7e61c9d3070c7dd282964f9f2bd6788a46a2f8f97de4209f"} Nov 02 08:10:24.892156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.892131 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" event={"ID":"1531694f-d551-47a5-93d8-d3efd6a87e36","Type":"ContainerStarted","Data":"b90e4510d903a1a64688a4f870356a034036b8767a04fc6f7585132b8c6c8af4"} Nov 02 08:10:24.894991 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.894943 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" event={"ID":"3b21ebb0-f617-4854-a2db-cf113a0eb2bf","Type":"ContainerStarted","Data":"45eb9a2c1061aa75efa5b0ec527bdefbdd72d329133966461003c54df987ebb5"} Nov 02 08:10:24.897505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.897472 6160 generic.go:356] "Generic (PLEG): container finished" podID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerID="c2cf332d06a9030f4ea29f7e9e929282a263dc652f8b8d2955bb55e7cfa9f316" exitCode=0 Nov 02 08:10:24.897607 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.897578 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" event={"ID":"c27216d9-90ea-4bc4-badd-c9a03ce61a57","Type":"ContainerDied","Data":"c2cf332d06a9030f4ea29f7e9e929282a263dc652f8b8d2955bb55e7cfa9f316"} Nov 02 08:10:24.900296 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.900277 6160 generic.go:356] "Generic (PLEG): container finished" podID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerID="c418f8ab48f5b3fd7039ba3328506b7df10b678843c6050ba6f187cb3d968575" exitCode=0 Nov 02 08:10:24.900393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.900364 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" event={"ID":"c3617eb7-9969-4430-bf56-0da8db1e9e28","Type":"ContainerDied","Data":"c418f8ab48f5b3fd7039ba3328506b7df10b678843c6050ba6f187cb3d968575"} Nov 02 08:10:24.902813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:24.902790 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" event={"ID":"2ee7da9e-0167-4577-bfc6-05921d3c0233","Type":"ContainerStarted","Data":"32638c5bb42bd4bcf2b9364ed43285e74a6408dbb9c8b522e6399b8787bccf3a"} Nov 02 08:10:25.213310 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.213209 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" podStartSLOduration=3.28630992 podStartE2EDuration="31.213167559s" totalImagesPullingTime="27.926857639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:25.212694935 +0000 UTC m=+223.766199195" watchObservedRunningTime="2026-11-02 08:10:25.213167559 +0000 UTC m=+223.766671756" Nov 02 08:10:25.549701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.549641 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" podStartSLOduration=3.89951306 podStartE2EDuration="31.549624288s" totalImagesPullingTime="27.650111228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:25.548370864 +0000 UTC m=+224.101875084" watchObservedRunningTime="2026-11-02 08:10:25.549624288 +0000 UTC m=+224.103128484" Nov 02 08:10:25.656721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.656660 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" podStartSLOduration=4.340315099 podStartE2EDuration="31.656640928s" totalImagesPullingTime="27.316325829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:25.655829803 +0000 UTC m=+224.209334005" watchObservedRunningTime="2026-11-02 08:10:25.656640928 +0000 UTC m=+224.210145126" Nov 02 08:10:25.915775 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.915737 6160 generic.go:356] "Generic (PLEG): container finished" podID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerID="32638c5bb42bd4bcf2b9364ed43285e74a6408dbb9c8b522e6399b8787bccf3a" exitCode=0 Nov 02 08:10:25.916270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.915810 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" event={"ID":"2ee7da9e-0167-4577-bfc6-05921d3c0233","Type":"ContainerDied","Data":"32638c5bb42bd4bcf2b9364ed43285e74a6408dbb9c8b522e6399b8787bccf3a"} Nov 02 08:10:25.918242 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.918209 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" event={"ID":"99ea3be8-4201-426e-aadd-609d23386998","Type":"ContainerStarted","Data":"bd32ad87f296bf7c2e06d90b939ed192e451be33a4a8a25f074eb0b7291c6a9a"} Nov 02 08:10:25.920564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.920541 6160 generic.go:356] "Generic (PLEG): container finished" podID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerID="67142b95a54d24740853aa3bf60896d94578281c972aa84d09e9a944c855b700" exitCode=0 Nov 02 08:10:25.920685 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.920613 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" event={"ID":"2b7cc0a4-6464-4d62-b076-e6dec51b41d3","Type":"ContainerDied","Data":"67142b95a54d24740853aa3bf60896d94578281c972aa84d09e9a944c855b700"} Nov 02 08:10:25.922973 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.922950 6160 generic.go:356] "Generic (PLEG): container finished" podID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerID="b90e4510d903a1a64688a4f870356a034036b8767a04fc6f7585132b8c6c8af4" exitCode=0 Nov 02 08:10:25.923073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.923010 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" event={"ID":"1531694f-d551-47a5-93d8-d3efd6a87e36","Type":"ContainerDied","Data":"b90e4510d903a1a64688a4f870356a034036b8767a04fc6f7585132b8c6c8af4"} Nov 02 08:10:25.925393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.925371 6160 generic.go:356] "Generic (PLEG): container finished" podID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerID="45eb9a2c1061aa75efa5b0ec527bdefbdd72d329133966461003c54df987ebb5" exitCode=0 Nov 02 08:10:25.925540 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.925446 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" event={"ID":"3b21ebb0-f617-4854-a2db-cf113a0eb2bf","Type":"ContainerDied","Data":"45eb9a2c1061aa75efa5b0ec527bdefbdd72d329133966461003c54df987ebb5"} Nov 02 08:10:25.927570 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.927550 6160 generic.go:356] "Generic (PLEG): container finished" podID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerID="4afa51c30a47f1930a19208a5a77063aeb96e5e9934008bc9151880e71a880eb" exitCode=0 Nov 02 08:10:25.927661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.927610 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" event={"ID":"92e17c7b-6cb5-4848-a75e-4e633a2297f0","Type":"ContainerDied","Data":"4afa51c30a47f1930a19208a5a77063aeb96e5e9934008bc9151880e71a880eb"} Nov 02 08:10:25.929836 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.929817 6160 generic.go:356] "Generic (PLEG): container finished" podID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerID="3bbe1db37f20beb06414d4f01530a70d90b91e57ccdc80d6347bfdc0a14f3b5a" exitCode=0 Nov 02 08:10:25.930152 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:25.930130 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" event={"ID":"27b82ea9-bab0-42fa-9d34-662bc2e89254","Type":"ContainerDied","Data":"3bbe1db37f20beb06414d4f01530a70d90b91e57ccdc80d6347bfdc0a14f3b5a"} Nov 02 08:10:26.151858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.151770 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" podStartSLOduration=3.285144911 podStartE2EDuration="34.151755048s" totalImagesPullingTime="30.866610137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:26.147606329 +0000 UTC m=+224.701110535" watchObservedRunningTime="2026-11-02 08:10:26.151755048 +0000 UTC m=+224.705259245" Nov 02 08:10:26.158623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.158599 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:10:26.161469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.161446 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:10:26.166557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.166537 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:10:26.174852 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.174832 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:10:26.176563 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.176543 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:10:26.273016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.272984 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3617eb7-9969-4430-bf56-0da8db1e9e28-kube-api-access-m5pqc\" (UniqueName: \"kubernetes.io/projected/c3617eb7-9969-4430-bf56-0da8db1e9e28-kube-api-access-m5pqc\") pod \"c3617eb7-9969-4430-bf56-0da8db1e9e28\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " Nov 02 08:10:26.273215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273046 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-bundle\") pod \"c3617eb7-9969-4430-bf56-0da8db1e9e28\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " Nov 02 08:10:26.273215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273097 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-bundle\") pod \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " Nov 02 08:10:26.273215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273121 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-bundle\") pod \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " Nov 02 08:10:26.273215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273160 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-kube-api-access-gkm5d\" (UniqueName: \"kubernetes.io/projected/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-kube-api-access-gkm5d\") pod \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " Nov 02 08:10:26.273380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273218 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-util\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-util\") pod \"c3617eb7-9969-4430-bf56-0da8db1e9e28\" (UID: \"c3617eb7-9969-4430-bf56-0da8db1e9e28\") " Nov 02 08:10:26.273380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273244 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-util\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-util\") pod \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\" (UID: \"a99d10da-d327-4799-b1c2-ef2c23cb5aa1\") " Nov 02 08:10:26.273380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273277 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-bundle\") pod \"cab83ba3-8d56-4572-badf-2c9f481faea6\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " Nov 02 08:10:26.273380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273325 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27216d9-90ea-4bc4-badd-c9a03ce61a57-kube-api-access-7kmgw\" (UniqueName: \"kubernetes.io/projected/c27216d9-90ea-4bc4-badd-c9a03ce61a57-kube-api-access-7kmgw\") pod \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " Nov 02 08:10:26.273380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273351 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-util\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-util\") pod \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\" (UID: \"c27216d9-90ea-4bc4-badd-c9a03ce61a57\") " Nov 02 08:10:26.273620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273378 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab83ba3-8d56-4572-badf-2c9f481faea6-kube-api-access-z754t\" (UniqueName: \"kubernetes.io/projected/cab83ba3-8d56-4572-badf-2c9f481faea6-kube-api-access-z754t\") pod \"cab83ba3-8d56-4572-badf-2c9f481faea6\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " Nov 02 08:10:26.273620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273436 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-util\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-util\") pod \"cab83ba3-8d56-4572-badf-2c9f481faea6\" (UID: \"cab83ba3-8d56-4572-badf-2c9f481faea6\") " Nov 02 08:10:26.273723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273611 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-bundle" pod "a99d10da-d327-4799-b1c2-ef2c23cb5aa1" (UID: "a99d10da-d327-4799-b1c2-ef2c23cb5aa1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:26.273723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.273713 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.274384 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.274346 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-bundle" pod "c27216d9-90ea-4bc4-badd-c9a03ce61a57" (UID: "c27216d9-90ea-4bc4-badd-c9a03ce61a57"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:26.275709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.275678 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-bundle" pod "cab83ba3-8d56-4572-badf-2c9f481faea6" (UID: "cab83ba3-8d56-4572-badf-2c9f481faea6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:26.276145 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.276118 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3617eb7-9969-4430-bf56-0da8db1e9e28-kube-api-access-m5pqc" pod "c3617eb7-9969-4430-bf56-0da8db1e9e28" (UID: "c3617eb7-9969-4430-bf56-0da8db1e9e28"). InnerVolumeSpecName "kube-api-access-m5pqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:26.276286 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.276244 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-bundle" pod "c3617eb7-9969-4430-bf56-0da8db1e9e28" (UID: "c3617eb7-9969-4430-bf56-0da8db1e9e28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:26.278614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.278585 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:26.280642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.280614 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-kube-api-access-gkm5d" pod "a99d10da-d327-4799-b1c2-ef2c23cb5aa1" (UID: "a99d10da-d327-4799-b1c2-ef2c23cb5aa1"). InnerVolumeSpecName "kube-api-access-gkm5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:26.280762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.280650 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27216d9-90ea-4bc4-badd-c9a03ce61a57-kube-api-access-7kmgw" pod "c27216d9-90ea-4bc4-badd-c9a03ce61a57" (UID: "c27216d9-90ea-4bc4-badd-c9a03ce61a57"). InnerVolumeSpecName "kube-api-access-7kmgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:26.281597 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.281576 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-util" pod "a99d10da-d327-4799-b1c2-ef2c23cb5aa1" (UID: "a99d10da-d327-4799-b1c2-ef2c23cb5aa1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:26.281702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.281677 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-util" pod "c27216d9-90ea-4bc4-badd-c9a03ce61a57" (UID: "c27216d9-90ea-4bc4-badd-c9a03ce61a57"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:26.282329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.282302 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab83ba3-8d56-4572-badf-2c9f481faea6-kube-api-access-z754t" pod "cab83ba3-8d56-4572-badf-2c9f481faea6" (UID: "cab83ba3-8d56-4572-badf-2c9f481faea6"). InnerVolumeSpecName "kube-api-access-z754t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:26.282438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.282414 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-util" pod "c3617eb7-9969-4430-bf56-0da8db1e9e28" (UID: "c3617eb7-9969-4430-bf56-0da8db1e9e28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:26.282916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.282894 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-util" pod "cab83ba3-8d56-4572-badf-2c9f481faea6" (UID: "cab83ba3-8d56-4572-badf-2c9f481faea6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:26.375031 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.374999 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375031 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375034 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkm5d\" (UniqueName: \"kubernetes.io/projected/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-kube-api-access-gkm5d\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375046 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375056 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a99d10da-d327-4799-b1c2-ef2c23cb5aa1-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375064 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375073 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kmgw\" (UniqueName: \"kubernetes.io/projected/c27216d9-90ea-4bc4-badd-c9a03ce61a57-kube-api-access-7kmgw\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375080 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c27216d9-90ea-4bc4-badd-c9a03ce61a57-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375088 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z754t\" (UniqueName: \"kubernetes.io/projected/cab83ba3-8d56-4572-badf-2c9f481faea6-kube-api-access-z754t\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375096 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cab83ba3-8d56-4572-badf-2c9f481faea6-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375105 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5pqc\" (UniqueName: \"kubernetes.io/projected/c3617eb7-9969-4430-bf56-0da8db1e9e28-kube-api-access-m5pqc\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.375257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.375113 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3617eb7-9969-4430-bf56-0da8db1e9e28-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:26.937404 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.937363 6160 generic.go:356] "Generic (PLEG): container finished" podID="99ea3be8-4201-426e-aadd-609d23386998" containerID="bd32ad87f296bf7c2e06d90b939ed192e451be33a4a8a25f074eb0b7291c6a9a" exitCode=0 Nov 02 08:10:26.937860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.937464 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" event={"ID":"99ea3be8-4201-426e-aadd-609d23386998","Type":"ContainerDied","Data":"bd32ad87f296bf7c2e06d90b939ed192e451be33a4a8a25f074eb0b7291c6a9a"} Nov 02 08:10:26.938975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.938953 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:10:26.939699 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.939683 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" Nov 02 08:10:26.939794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.939705 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz" event={"ID":"a99d10da-d327-4799-b1c2-ef2c23cb5aa1","Type":"ContainerDied","Data":"edf3716c483c3878744ee7dccd48d9c79a4cb86389f9f8825046f54e1e588e29"} Nov 02 08:10:26.939794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.939746 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf3716c483c3878744ee7dccd48d9c79a4cb86389f9f8825046f54e1e588e29" Nov 02 08:10:26.941795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.941775 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" Nov 02 08:10:26.941893 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.941802 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zkgc7" event={"ID":"cab83ba3-8d56-4572-badf-2c9f481faea6","Type":"ContainerDied","Data":"34d3a9a6b93ca66fc49cd623363c8622024c9674acc711ede876fca61adb6ac3"} Nov 02 08:10:26.941893 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.941849 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34d3a9a6b93ca66fc49cd623363c8622024c9674acc711ede876fca61adb6ac3" Nov 02 08:10:26.943993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.943932 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" event={"ID":"c27216d9-90ea-4bc4-badd-c9a03ce61a57","Type":"ContainerDied","Data":"7db6625827aa328a5f17e27d35f88da500e85872a81e5c7d1cd4a038fcb9ffda"} Nov 02 08:10:26.943993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.943949 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axhrw6" Nov 02 08:10:26.943993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.943970 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7db6625827aa328a5f17e27d35f88da500e85872a81e5c7d1cd4a038fcb9ffda" Nov 02 08:10:26.946784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.946382 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" event={"ID":"102f9b5d-cabe-4703-befb-33eeda727d34","Type":"ContainerStarted","Data":"4207a124b9a596b114ed81a009976b4984e92cb51714cb441e63e5df3510f4c5"} Nov 02 08:10:26.949165 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.949142 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" Nov 02 08:10:26.949759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.949689 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kn28n" event={"ID":"c3617eb7-9969-4430-bf56-0da8db1e9e28","Type":"ContainerDied","Data":"15d3bd5b0e391c067913c2b1ad3d5b96f17363a2569d0b0c526aad41132e893e"} Nov 02 08:10:26.949759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:26.949728 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15d3bd5b0e391c067913c2b1ad3d5b96f17363a2569d0b0c526aad41132e893e" Nov 02 08:10:27.124125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.124092 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:10:27.169777 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.169750 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:10:27.177615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.177511 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:10:27.179837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.179811 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:10:27.182428 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.182401 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-bundle\") pod \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " Nov 02 08:10:27.182532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.182484 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-util\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-util\") pod \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " Nov 02 08:10:27.182588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.182543 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-kube-api-access-lgnqt\" (UniqueName: \"kubernetes.io/projected/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-kube-api-access-lgnqt\") pod \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\" (UID: \"2b7cc0a4-6464-4d62-b076-e6dec51b41d3\") " Nov 02 08:10:27.182886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.182860 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-bundle" pod "2b7cc0a4-6464-4d62-b076-e6dec51b41d3" (UID: "2b7cc0a4-6464-4d62-b076-e6dec51b41d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.185079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.185052 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:10:27.185758 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.185738 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-kube-api-access-lgnqt" pod "2b7cc0a4-6464-4d62-b076-e6dec51b41d3" (UID: "2b7cc0a4-6464-4d62-b076-e6dec51b41d3"). InnerVolumeSpecName "kube-api-access-lgnqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:27.188054 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.188005 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:10:27.188647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.188626 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-util" pod "2b7cc0a4-6464-4d62-b076-e6dec51b41d3" (UID: "2b7cc0a4-6464-4d62-b076-e6dec51b41d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.283615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283580 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-util\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-util\") pod \"27b82ea9-bab0-42fa-9d34-662bc2e89254\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " Nov 02 08:10:27.283806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283634 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-bundle\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-bundle\") pod \"27b82ea9-bab0-42fa-9d34-662bc2e89254\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " Nov 02 08:10:27.283806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283658 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-util\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-util\") pod \"1531694f-d551-47a5-93d8-d3efd6a87e36\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " Nov 02 08:10:27.283806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283684 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1531694f-d551-47a5-93d8-d3efd6a87e36-kube-api-access-jbr6v\" (UniqueName: \"kubernetes.io/projected/1531694f-d551-47a5-93d8-d3efd6a87e36-kube-api-access-jbr6v\") pod \"1531694f-d551-47a5-93d8-d3efd6a87e36\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " Nov 02 08:10:27.283806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283720 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-bundle\") pod \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " Nov 02 08:10:27.283806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283757 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ee7da9e-0167-4577-bfc6-05921d3c0233-kube-api-access-89dwd\" (UniqueName: \"kubernetes.io/projected/2ee7da9e-0167-4577-bfc6-05921d3c0233-kube-api-access-89dwd\") pod \"2ee7da9e-0167-4577-bfc6-05921d3c0233\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " Nov 02 08:10:27.283806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283776 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-bundle\") pod \"2ee7da9e-0167-4577-bfc6-05921d3c0233\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " Nov 02 08:10:27.284137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283824 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-util\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-util\") pod \"2ee7da9e-0167-4577-bfc6-05921d3c0233\" (UID: \"2ee7da9e-0167-4577-bfc6-05921d3c0233\") " Nov 02 08:10:27.284137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283857 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-util\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-util\") pod \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " Nov 02 08:10:27.284137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283891 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b82ea9-bab0-42fa-9d34-662bc2e89254-kube-api-access-nff9k\" (UniqueName: \"kubernetes.io/projected/27b82ea9-bab0-42fa-9d34-662bc2e89254-kube-api-access-nff9k\") pod \"27b82ea9-bab0-42fa-9d34-662bc2e89254\" (UID: \"27b82ea9-bab0-42fa-9d34-662bc2e89254\") " Nov 02 08:10:27.284137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283955 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-kube-api-access-dk8w5\" (UniqueName: \"kubernetes.io/projected/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-kube-api-access-dk8w5\") pod \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " Nov 02 08:10:27.284137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.283998 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-util\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-util\") pod \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\" (UID: \"3b21ebb0-f617-4854-a2db-cf113a0eb2bf\") " Nov 02 08:10:27.284137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.284031 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-bundle\") pod \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " Nov 02 08:10:27.284137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.284072 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e17c7b-6cb5-4848-a75e-4e633a2297f0-kube-api-access-jgrdq\" (UniqueName: \"kubernetes.io/projected/92e17c7b-6cb5-4848-a75e-4e633a2297f0-kube-api-access-jgrdq\") pod \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\" (UID: \"92e17c7b-6cb5-4848-a75e-4e633a2297f0\") " Nov 02 08:10:27.284137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.284118 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-bundle\") pod \"1531694f-d551-47a5-93d8-d3efd6a87e36\" (UID: \"1531694f-d551-47a5-93d8-d3efd6a87e36\") " Nov 02 08:10:27.284548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.284393 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.284548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.284413 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgnqt\" (UniqueName: \"kubernetes.io/projected/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-kube-api-access-lgnqt\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.284548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.284429 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7cc0a4-6464-4d62-b076-e6dec51b41d3-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.289113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.284902 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-bundle" pod "2ee7da9e-0167-4577-bfc6-05921d3c0233" (UID: "2ee7da9e-0167-4577-bfc6-05921d3c0233"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.289113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.285701 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-bundle" pod "92e17c7b-6cb5-4848-a75e-4e633a2297f0" (UID: "92e17c7b-6cb5-4848-a75e-4e633a2297f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.289356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.289330 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-bundle" pod "3b21ebb0-f617-4854-a2db-cf113a0eb2bf" (UID: "3b21ebb0-f617-4854-a2db-cf113a0eb2bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.289632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.289607 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-bundle" pod "1531694f-d551-47a5-93d8-d3efd6a87e36" (UID: "1531694f-d551-47a5-93d8-d3efd6a87e36"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.290442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.290416 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-bundle" pod "27b82ea9-bab0-42fa-9d34-662bc2e89254" (UID: "27b82ea9-bab0-42fa-9d34-662bc2e89254"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.290442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.290424 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e17c7b-6cb5-4848-a75e-4e633a2297f0-kube-api-access-jgrdq" pod "92e17c7b-6cb5-4848-a75e-4e633a2297f0" (UID: "92e17c7b-6cb5-4848-a75e-4e633a2297f0"). InnerVolumeSpecName "kube-api-access-jgrdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:27.291255 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.291232 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b82ea9-bab0-42fa-9d34-662bc2e89254-kube-api-access-nff9k" pod "27b82ea9-bab0-42fa-9d34-662bc2e89254" (UID: "27b82ea9-bab0-42fa-9d34-662bc2e89254"). InnerVolumeSpecName "kube-api-access-nff9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:27.293434 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.293410 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-util" pod "3b21ebb0-f617-4854-a2db-cf113a0eb2bf" (UID: "3b21ebb0-f617-4854-a2db-cf113a0eb2bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.293509 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.293482 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-util" pod "27b82ea9-bab0-42fa-9d34-662bc2e89254" (UID: "27b82ea9-bab0-42fa-9d34-662bc2e89254"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.293802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.293779 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-util" pod "92e17c7b-6cb5-4848-a75e-4e633a2297f0" (UID: "92e17c7b-6cb5-4848-a75e-4e633a2297f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.294009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.293981 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1531694f-d551-47a5-93d8-d3efd6a87e36-kube-api-access-jbr6v" pod "1531694f-d551-47a5-93d8-d3efd6a87e36" (UID: "1531694f-d551-47a5-93d8-d3efd6a87e36"). InnerVolumeSpecName "kube-api-access-jbr6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:27.294009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.293985 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee7da9e-0167-4577-bfc6-05921d3c0233-kube-api-access-89dwd" pod "2ee7da9e-0167-4577-bfc6-05921d3c0233" (UID: "2ee7da9e-0167-4577-bfc6-05921d3c0233"). InnerVolumeSpecName "kube-api-access-89dwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:27.294149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.294038 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-util" pod "1531694f-d551-47a5-93d8-d3efd6a87e36" (UID: "1531694f-d551-47a5-93d8-d3efd6a87e36"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.294941 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.294922 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-util" pod "2ee7da9e-0167-4577-bfc6-05921d3c0233" (UID: "2ee7da9e-0167-4577-bfc6-05921d3c0233"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:27.295112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.295099 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-kube-api-access-dk8w5" pod "3b21ebb0-f617-4854-a2db-cf113a0eb2bf" (UID: "3b21ebb0-f617-4854-a2db-cf113a0eb2bf"). InnerVolumeSpecName "kube-api-access-dk8w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:27.385294 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385258 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbr6v\" (UniqueName: \"kubernetes.io/projected/1531694f-d551-47a5-93d8-d3efd6a87e36-kube-api-access-jbr6v\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385294 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385288 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385294 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385297 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dwd\" (UniqueName: \"kubernetes.io/projected/2ee7da9e-0167-4577-bfc6-05921d3c0233-kube-api-access-89dwd\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385307 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385317 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ee7da9e-0167-4577-bfc6-05921d3c0233-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385325 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385333 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nff9k\" (UniqueName: \"kubernetes.io/projected/27b82ea9-bab0-42fa-9d34-662bc2e89254-kube-api-access-nff9k\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385341 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8w5\" (UniqueName: \"kubernetes.io/projected/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-kube-api-access-dk8w5\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385351 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b21ebb0-f617-4854-a2db-cf113a0eb2bf-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385358 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e17c7b-6cb5-4848-a75e-4e633a2297f0-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385367 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrdq\" (UniqueName: \"kubernetes.io/projected/92e17c7b-6cb5-4848-a75e-4e633a2297f0-kube-api-access-jgrdq\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385375 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385383 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385390 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27b82ea9-bab0-42fa-9d34-662bc2e89254-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.385528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.385397 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1531694f-d551-47a5-93d8-d3efd6a87e36-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:27.556923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.556864 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" podStartSLOduration=3.467567311 podStartE2EDuration="45.556852273s" totalImagesPullingTime="42.089284962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:27.556531178 +0000 UTC m=+226.110035375" watchObservedRunningTime="2026-11-02 08:10:27.556852273 +0000 UTC m=+226.110356522" Nov 02 08:10:27.845327 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.845245 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:27.958308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.958281 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" Nov 02 08:10:27.958783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.958272 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wqph" event={"ID":"2b7cc0a4-6464-4d62-b076-e6dec51b41d3","Type":"ContainerDied","Data":"cefee873a2f861248273735a789018ae8bd0714f2b0abd6289b095f88c02fdba"} Nov 02 08:10:27.958783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.958420 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cefee873a2f861248273735a789018ae8bd0714f2b0abd6289b095f88c02fdba" Nov 02 08:10:27.960851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.960823 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" event={"ID":"1531694f-d551-47a5-93d8-d3efd6a87e36","Type":"ContainerDied","Data":"79884a5c0b2d8043fd732d49909600f59dbcdb9f5a0a9c17c49505fc0f67ff61"} Nov 02 08:10:27.960989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.960866 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79884a5c0b2d8043fd732d49909600f59dbcdb9f5a0a9c17c49505fc0f67ff61" Nov 02 08:10:27.960989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.960947 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hbrh6" Nov 02 08:10:27.963533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.963505 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" event={"ID":"3b21ebb0-f617-4854-a2db-cf113a0eb2bf","Type":"ContainerDied","Data":"b8440c38406fadf4cf89f64d3e696272103a68b820295c98ba248d318b0e7991"} Nov 02 08:10:27.963646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.963542 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a86frc" Nov 02 08:10:27.963646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.963544 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8440c38406fadf4cf89f64d3e696272103a68b820295c98ba248d318b0e7991" Nov 02 08:10:27.965634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.965614 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" Nov 02 08:10:27.965759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.965613 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdx2h" event={"ID":"92e17c7b-6cb5-4848-a75e-4e633a2297f0","Type":"ContainerDied","Data":"aa36e6e46dbc1802b9d34b5a6ef72aad8d71abb3fb338d48bc1461a0528f6477"} Nov 02 08:10:27.965833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.965765 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa36e6e46dbc1802b9d34b5a6ef72aad8d71abb3fb338d48bc1461a0528f6477" Nov 02 08:10:27.967871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.967849 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" event={"ID":"27b82ea9-bab0-42fa-9d34-662bc2e89254","Type":"ContainerDied","Data":"bd97e10394b4cbed7fbbc9924bcd1a3f9e7a73009e48e8e5e3fb5628e37e7574"} Nov 02 08:10:27.967978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.967884 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd97e10394b4cbed7fbbc9924bcd1a3f9e7a73009e48e8e5e3fb5628e37e7574" Nov 02 08:10:27.967978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.967962 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxmw88" Nov 02 08:10:27.970269 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.970249 6160 generic.go:356] "Generic (PLEG): container finished" podID="102f9b5d-cabe-4703-befb-33eeda727d34" containerID="4207a124b9a596b114ed81a009976b4984e92cb51714cb441e63e5df3510f4c5" exitCode=0 Nov 02 08:10:27.970355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.970338 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" event={"ID":"102f9b5d-cabe-4703-befb-33eeda727d34","Type":"ContainerDied","Data":"4207a124b9a596b114ed81a009976b4984e92cb51714cb441e63e5df3510f4c5"} Nov 02 08:10:27.972189 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.972160 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" Nov 02 08:10:27.972273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.972195 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnzjsr" event={"ID":"2ee7da9e-0167-4577-bfc6-05921d3c0233","Type":"ContainerDied","Data":"777c95ba72d21a28da5977c1e68682602841c2765b83798b1a44df1c5ef5fbdc"} Nov 02 08:10:27.972273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:27.972231 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="777c95ba72d21a28da5977c1e68682602841c2765b83798b1a44df1c5ef5fbdc" Nov 02 08:10:28.101888 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.101836 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:10:28.122845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.122809 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:28.193020 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.192994 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-util\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-util\") pod \"99ea3be8-4201-426e-aadd-609d23386998\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " Nov 02 08:10:28.193155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.193030 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ea3be8-4201-426e-aadd-609d23386998-kube-api-access-mfhsr\" (UniqueName: \"kubernetes.io/projected/99ea3be8-4201-426e-aadd-609d23386998-kube-api-access-mfhsr\") pod \"99ea3be8-4201-426e-aadd-609d23386998\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " Nov 02 08:10:28.193155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.193091 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-bundle\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-bundle\") pod \"99ea3be8-4201-426e-aadd-609d23386998\" (UID: \"99ea3be8-4201-426e-aadd-609d23386998\") " Nov 02 08:10:28.193681 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.193657 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-bundle" pod "99ea3be8-4201-426e-aadd-609d23386998" (UID: "99ea3be8-4201-426e-aadd-609d23386998"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:28.195432 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.195406 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ea3be8-4201-426e-aadd-609d23386998-kube-api-access-mfhsr" pod "99ea3be8-4201-426e-aadd-609d23386998" (UID: "99ea3be8-4201-426e-aadd-609d23386998"). InnerVolumeSpecName "kube-api-access-mfhsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:28.200510 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.200486 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-util" pod "99ea3be8-4201-426e-aadd-609d23386998" (UID: "99ea3be8-4201-426e-aadd-609d23386998"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:28.293795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.293767 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:28.293795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.293793 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfhsr\" (UniqueName: \"kubernetes.io/projected/99ea3be8-4201-426e-aadd-609d23386998-kube-api-access-mfhsr\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:28.293958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.293804 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99ea3be8-4201-426e-aadd-609d23386998-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:28.681153 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681119 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl"] Nov 02 08:10:28.681396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681383 6160 cpu_manager.go:414] "containerMap: removing container" podUID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerName="util" Nov 02 08:10:28.681437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681397 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerName="util" Nov 02 08:10:28.681437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681408 6160 cpu_manager.go:414] "containerMap: removing container" podUID="99ea3be8-4201-426e-aadd-609d23386998" containerName="util" Nov 02 08:10:28.681437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681413 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ea3be8-4201-426e-aadd-609d23386998" containerName="util" Nov 02 08:10:28.681437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681421 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerName="util" Nov 02 08:10:28.681437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681426 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerName="util" Nov 02 08:10:28.681437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681432 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerName="pull" Nov 02 08:10:28.681437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681437 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681444 6160 cpu_manager.go:414] "containerMap: removing container" podUID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681448 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681455 6160 cpu_manager.go:414] "containerMap: removing container" podUID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681460 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681467 6160 cpu_manager.go:414] "containerMap: removing container" podUID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681471 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681478 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681482 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681491 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681494 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681507 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681511 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681518 6160 cpu_manager.go:414] "containerMap: removing container" podUID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681522 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681528 6160 cpu_manager.go:414] "containerMap: removing container" podUID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681532 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681538 6160 cpu_manager.go:414] "containerMap: removing container" podUID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681542 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681548 6160 cpu_manager.go:414] "containerMap: removing container" podUID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681552 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681557 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681562 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerName="pull" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681569 6160 cpu_manager.go:414] "containerMap: removing container" podUID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681573 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681579 6160 cpu_manager.go:414] "containerMap: removing container" podUID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681583 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681588 6160 cpu_manager.go:414] "containerMap: removing container" podUID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681592 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681597 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681600 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681606 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681610 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerName="util" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681619 6160 cpu_manager.go:414] "containerMap: removing container" podUID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681624 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681630 6160 cpu_manager.go:414] "containerMap: removing container" podUID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681634 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681641 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681645 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681650 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerName="extract" Nov 02 08:10:28.681639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681654 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681662 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerName="util" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681666 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerName="util" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681672 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerName="pull" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681676 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerName="pull" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681683 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerName="pull" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681686 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerName="pull" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681692 6160 cpu_manager.go:414] "containerMap: removing container" podUID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681696 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681703 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerName="util" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681706 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerName="util" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681711 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681715 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681722 6160 cpu_manager.go:414] "containerMap: removing container" podUID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerName="pull" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681726 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerName="pull" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681732 6160 cpu_manager.go:414] "containerMap: removing container" podUID="99ea3be8-4201-426e-aadd-609d23386998" containerName="pull" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681735 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ea3be8-4201-426e-aadd-609d23386998" containerName="pull" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681742 6160 cpu_manager.go:414] "containerMap: removing container" podUID="99ea3be8-4201-426e-aadd-609d23386998" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681746 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ea3be8-4201-426e-aadd-609d23386998" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681782 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e17c7b-6cb5-4848-a75e-4e633a2297f0" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681791 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3617eb7-9969-4430-bf56-0da8db1e9e28" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681797 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b21ebb0-f617-4854-a2db-cf113a0eb2bf" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681807 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27216d9-90ea-4bc4-badd-c9a03ce61a57" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681812 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab83ba3-8d56-4572-badf-2c9f481faea6" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681817 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b82ea9-bab0-42fa-9d34-662bc2e89254" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681822 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ee7da9e-0167-4577-bfc6-05921d3c0233" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681827 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7cc0a4-6464-4d62-b076-e6dec51b41d3" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681833 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="1531694f-d551-47a5-93d8-d3efd6a87e36" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681838 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99d10da-d327-4799-b1c2-ef2c23cb5aa1" containerName="extract" Nov 02 08:10:28.682694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.681844 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ea3be8-4201-426e-aadd-609d23386998" containerName="extract" Nov 02 08:10:28.683853 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.683837 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:28.732522 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.732491 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 08:10:28.732522 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.732519 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 08:10:28.732693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.732562 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5fslj\"" Nov 02 08:10:28.738984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.738961 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl"] Nov 02 08:10:28.797523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.797372 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8kc\" (UniqueName: \"kubernetes.io/projected/a70658ef-7918-4958-baeb-03bb7bc96e1a-kube-api-access-rr8kc\") pod \"cert-manager-webhook-5dbbbf75cc-5zfvl\" (UID: \"a70658ef-7918-4958-baeb-03bb7bc96e1a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:28.797523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.797414 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a70658ef-7918-4958-baeb-03bb7bc96e1a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5zfvl\" (UID: \"a70658ef-7918-4958-baeb-03bb7bc96e1a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:28.898135 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.898103 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a70658ef-7918-4958-baeb-03bb7bc96e1a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5zfvl\" (UID: \"a70658ef-7918-4958-baeb-03bb7bc96e1a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:28.898303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.898163 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8kc\" (UniqueName: \"kubernetes.io/projected/a70658ef-7918-4958-baeb-03bb7bc96e1a-kube-api-access-rr8kc\") pod \"cert-manager-webhook-5dbbbf75cc-5zfvl\" (UID: \"a70658ef-7918-4958-baeb-03bb7bc96e1a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:28.958272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.958204 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a70658ef-7918-4958-baeb-03bb7bc96e1a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5zfvl\" (UID: \"a70658ef-7918-4958-baeb-03bb7bc96e1a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:28.958403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.958346 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8kc\" (UniqueName: \"kubernetes.io/projected/a70658ef-7918-4958-baeb-03bb7bc96e1a-kube-api-access-rr8kc\") pod \"cert-manager-webhook-5dbbbf75cc-5zfvl\" (UID: \"a70658ef-7918-4958-baeb-03bb7bc96e1a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:28.979150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.979121 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" Nov 02 08:10:28.979307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.979144 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjs7z" event={"ID":"99ea3be8-4201-426e-aadd-609d23386998","Type":"ContainerDied","Data":"bb4cb9ebf03f085dc35cf5bc45c783b5e3ff9891e969367d4c3b1ca7afcd9dc8"} Nov 02 08:10:28.979307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:28.979267 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb4cb9ebf03f085dc35cf5bc45c783b5e3ff9891e969367d4c3b1ca7afcd9dc8" Nov 02 08:10:29.008751 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.008415 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:29.104779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.104754 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:10:29.200978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.200943 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102f9b5d-cabe-4703-befb-33eeda727d34-kube-api-access-pgnf2\" (UniqueName: \"kubernetes.io/projected/102f9b5d-cabe-4703-befb-33eeda727d34-kube-api-access-pgnf2\") pod \"102f9b5d-cabe-4703-befb-33eeda727d34\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " Nov 02 08:10:29.201167 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.201006 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-bundle\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-bundle\") pod \"102f9b5d-cabe-4703-befb-33eeda727d34\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " Nov 02 08:10:29.201167 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.201033 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-util\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-util\") pod \"102f9b5d-cabe-4703-befb-33eeda727d34\" (UID: \"102f9b5d-cabe-4703-befb-33eeda727d34\") " Nov 02 08:10:29.201688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.201663 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-bundle" pod "102f9b5d-cabe-4703-befb-33eeda727d34" (UID: "102f9b5d-cabe-4703-befb-33eeda727d34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:29.203124 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.203098 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102f9b5d-cabe-4703-befb-33eeda727d34-kube-api-access-pgnf2" pod "102f9b5d-cabe-4703-befb-33eeda727d34" (UID: "102f9b5d-cabe-4703-befb-33eeda727d34"). InnerVolumeSpecName "kube-api-access-pgnf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:29.206546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.206523 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-util" pod "102f9b5d-cabe-4703-befb-33eeda727d34" (UID: "102f9b5d-cabe-4703-befb-33eeda727d34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:29.302512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.302426 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgnf2\" (UniqueName: \"kubernetes.io/projected/102f9b5d-cabe-4703-befb-33eeda727d34-kube-api-access-pgnf2\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:29.302512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.302458 6160 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-bundle\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:29.302512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.302470 6160 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/102f9b5d-cabe-4703-befb-33eeda727d34-util\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:10:29.332399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.332362 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:29.375293 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.375262 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl"] Nov 02 08:10:29.378198 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:10:29.378153 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda70658ef_7918_4958_baeb_03bb7bc96e1a.slice/crio-816d36ba852d52c13778f291719b06071839ff9320d469c7ab4118ff0a306162 WatchSource:0}: Error finding container 816d36ba852d52c13778f291719b06071839ff9320d469c7ab4118ff0a306162: Status 404 returned error can't find the container with id 816d36ba852d52c13778f291719b06071839ff9320d469c7ab4118ff0a306162 Nov 02 08:10:29.504395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.504360 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:29.984522 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.984487 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" event={"ID":"a70658ef-7918-4958-baeb-03bb7bc96e1a","Type":"ContainerStarted","Data":"816d36ba852d52c13778f291719b06071839ff9320d469c7ab4118ff0a306162"} Nov 02 08:10:29.986347 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.986318 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" event={"ID":"102f9b5d-cabe-4703-befb-33eeda727d34","Type":"ContainerDied","Data":"b2513fec4b7ccd13d50797324031e677084418db49022a7847815858a2997072"} Nov 02 08:10:29.986347 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.986349 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnztqf" Nov 02 08:10:29.986502 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:29.986367 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2513fec4b7ccd13d50797324031e677084418db49022a7847815858a2997072" Nov 02 08:10:30.198183 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.198151 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:30.442086 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.442049 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z"] Nov 02 08:10:30.442366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.442353 6160 cpu_manager.go:414] "containerMap: removing container" podUID="102f9b5d-cabe-4703-befb-33eeda727d34" containerName="util" Nov 02 08:10:30.442414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.442370 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102f9b5d-cabe-4703-befb-33eeda727d34" containerName="util" Nov 02 08:10:30.442414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.442393 6160 cpu_manager.go:414] "containerMap: removing container" podUID="102f9b5d-cabe-4703-befb-33eeda727d34" containerName="pull" Nov 02 08:10:30.442414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.442398 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102f9b5d-cabe-4703-befb-33eeda727d34" containerName="pull" Nov 02 08:10:30.442414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.442407 6160 cpu_manager.go:414] "containerMap: removing container" podUID="102f9b5d-cabe-4703-befb-33eeda727d34" containerName="extract" Nov 02 08:10:30.442414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.442412 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102f9b5d-cabe-4703-befb-33eeda727d34" containerName="extract" Nov 02 08:10:30.442559 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.442481 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="102f9b5d-cabe-4703-befb-33eeda727d34" containerName="extract" Nov 02 08:10:30.444794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.444777 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" Nov 02 08:10:30.493010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.492978 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tjc7z\"" Nov 02 08:10:30.510469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.510443 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e62f8b7d-622f-4882-8c0a-7d54fbe65778-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jph7z\" (UID: \"e62f8b7d-622f-4882-8c0a-7d54fbe65778\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" Nov 02 08:10:30.510658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.510504 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j2bs\" (UniqueName: \"kubernetes.io/projected/e62f8b7d-622f-4882-8c0a-7d54fbe65778-kube-api-access-5j2bs\") pod \"cert-manager-cainjector-5c57bbd4fc-jph7z\" (UID: \"e62f8b7d-622f-4882-8c0a-7d54fbe65778\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" Nov 02 08:10:30.540380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.540350 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z"] Nov 02 08:10:30.611355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.611328 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e62f8b7d-622f-4882-8c0a-7d54fbe65778-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jph7z\" (UID: \"e62f8b7d-622f-4882-8c0a-7d54fbe65778\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" Nov 02 08:10:30.611509 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.611412 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j2bs\" (UniqueName: \"kubernetes.io/projected/e62f8b7d-622f-4882-8c0a-7d54fbe65778-kube-api-access-5j2bs\") pod \"cert-manager-cainjector-5c57bbd4fc-jph7z\" (UID: \"e62f8b7d-622f-4882-8c0a-7d54fbe65778\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" Nov 02 08:10:30.665582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.665548 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e62f8b7d-622f-4882-8c0a-7d54fbe65778-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jph7z\" (UID: \"e62f8b7d-622f-4882-8c0a-7d54fbe65778\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" Nov 02 08:10:30.665731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.665619 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j2bs\" (UniqueName: \"kubernetes.io/projected/e62f8b7d-622f-4882-8c0a-7d54fbe65778-kube-api-access-5j2bs\") pod \"cert-manager-cainjector-5c57bbd4fc-jph7z\" (UID: \"e62f8b7d-622f-4882-8c0a-7d54fbe65778\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" Nov 02 08:10:30.690654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.690618 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:30.754949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:30.754872 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" Nov 02 08:10:31.113947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:31.113867 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z"] Nov 02 08:10:31.116682 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:10:31.116654 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode62f8b7d_622f_4882_8c0a_7d54fbe65778.slice/crio-5cf91fc0edd7e22950ef540e6fbd75336ce30a177cce66f9b5ee62993741f68d WatchSource:0}: Error finding container 5cf91fc0edd7e22950ef540e6fbd75336ce30a177cce66f9b5ee62993741f68d: Status 404 returned error can't find the container with id 5cf91fc0edd7e22950ef540e6fbd75336ce30a177cce66f9b5ee62993741f68d Nov 02 08:10:31.997664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:31.997629 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" event={"ID":"e62f8b7d-622f-4882-8c0a-7d54fbe65778","Type":"ContainerStarted","Data":"5cf91fc0edd7e22950ef540e6fbd75336ce30a177cce66f9b5ee62993741f68d"} Nov 02 08:10:33.325524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:33.325476 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:33.565812 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:33.565715 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:34.008729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.008692 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" event={"ID":"e62f8b7d-622f-4882-8c0a-7d54fbe65778","Type":"ContainerStarted","Data":"86a1f428126e921ba4f66a8f90b4be4f73d4b514ad16bccde963a81cc71a5175"} Nov 02 08:10:34.010382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.010358 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" event={"ID":"a70658ef-7918-4958-baeb-03bb7bc96e1a","Type":"ContainerStarted","Data":"ef8d7ffd52640f8c78b9aa9597dea010b61c01e6465fd4774eca53e8dc65054a"} Nov 02 08:10:34.010496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.010403 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:34.139096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.139031 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jph7z" podStartSLOduration=1.3384715090000001 podStartE2EDuration="4.139016682s" totalImagesPullingTime="2.800545173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:34.137496442 +0000 UTC m=+232.691000657" watchObservedRunningTime="2026-11-02 08:10:34.139016682 +0000 UTC m=+232.692520881" Nov 02 08:10:34.252550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.252424 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" podStartSLOduration=2.410355252 podStartE2EDuration="6.252408048s" totalImagesPullingTime="3.842052796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:34.251421086 +0000 UTC m=+232.804925303" watchObservedRunningTime="2026-11-02 08:10:34.252408048 +0000 UTC m=+232.805912245" Nov 02 08:10:34.503934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.503820 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:34.778549 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.778449 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw"] Nov 02 08:10:34.780705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.780688 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:34.896874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.896837 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw"] Nov 02 08:10:34.901406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.901374 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 08:10:34.901406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.901378 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-x42xc\"" Nov 02 08:10:34.901623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.901418 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 08:10:34.901623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.901422 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 08:10:34.901623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.901463 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 08:10:34.901826 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.901803 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 08:10:34.947832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.947798 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdnfm\" (UniqueName: \"kubernetes.io/projected/0dd118ab-44e1-4bd2-95d7-157d2564c23c-kube-api-access-qdnfm\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:34.948003 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.947849 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0dd118ab-44e1-4bd2-95d7-157d2564c23c-metrics-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:34.948003 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.947928 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0dd118ab-44e1-4bd2-95d7-157d2564c23c-manifests\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:34.948003 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.947986 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dd118ab-44e1-4bd2-95d7-157d2564c23c-webhook-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:34.948135 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.948011 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0dd118ab-44e1-4bd2-95d7-157d2564c23c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:34.948135 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:34.948037 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dd118ab-44e1-4bd2-95d7-157d2564c23c-tmp\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.018074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.018037 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:35.049210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.049116 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdnfm\" (UniqueName: \"kubernetes.io/projected/0dd118ab-44e1-4bd2-95d7-157d2564c23c-kube-api-access-qdnfm\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.049210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.049194 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0dd118ab-44e1-4bd2-95d7-157d2564c23c-metrics-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.049398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.049223 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0dd118ab-44e1-4bd2-95d7-157d2564c23c-manifests\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.049398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.049245 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dd118ab-44e1-4bd2-95d7-157d2564c23c-webhook-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.049398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.049272 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0dd118ab-44e1-4bd2-95d7-157d2564c23c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.049398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.049300 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dd118ab-44e1-4bd2-95d7-157d2564c23c-tmp\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.049701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.049670 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0dd118ab-44e1-4bd2-95d7-157d2564c23c-manifests\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.051792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.051767 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dd118ab-44e1-4bd2-95d7-157d2564c23c-tmp\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.051996 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.051976 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0dd118ab-44e1-4bd2-95d7-157d2564c23c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.052038 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.052002 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dd118ab-44e1-4bd2-95d7-157d2564c23c-webhook-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.052110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.052094 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0dd118ab-44e1-4bd2-95d7-157d2564c23c-metrics-cert\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.103127 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.103094 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdnfm\" (UniqueName: \"kubernetes.io/projected/0dd118ab-44e1-4bd2-95d7-157d2564c23c-kube-api-access-qdnfm\") pod \"opendatahub-operator-controller-manager-b679c64f6-l58kw\" (UID: \"0dd118ab-44e1-4bd2-95d7-157d2564c23c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.237520 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.237482 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:35.394916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.394826 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:35.430311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.430271 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:35.920065 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:10:35.920029 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd118ab_44e1_4bd2_95d7_157d2564c23c.slice/crio-d00213763b082daca0b6d0e60f4424cd725f37025f685f7022757daf46da0f55 WatchSource:0}: Error finding container d00213763b082daca0b6d0e60f4424cd725f37025f685f7022757daf46da0f55: Status 404 returned error can't find the container with id d00213763b082daca0b6d0e60f4424cd725f37025f685f7022757daf46da0f55 Nov 02 08:10:35.964319 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:35.964282 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw"] Nov 02 08:10:36.020626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:36.020590 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" event={"ID":"0dd118ab-44e1-4bd2-95d7-157d2564c23c","Type":"ContainerStarted","Data":"d00213763b082daca0b6d0e60f4424cd725f37025f685f7022757daf46da0f55"} Nov 02 08:10:36.434970 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:36.434934 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:36.655622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:36.655586 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:37.929035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:37.929006 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:38.104534 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:38.104494 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:38.660994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:38.660958 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:40.017147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:40.017114 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5zfvl" Nov 02 08:10:40.043476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:40.043443 6160 generic.go:356] "Generic (PLEG): container finished" podID="0dd118ab-44e1-4bd2-95d7-157d2564c23c" containerID="ff355f0af756d3e0877acedfdb7b2f62258e217606682baff1ef90591fd48801" exitCode=0 Nov 02 08:10:40.043611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:40.043528 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" event={"ID":"0dd118ab-44e1-4bd2-95d7-157d2564c23c","Type":"ContainerDied","Data":"ff355f0af756d3e0877acedfdb7b2f62258e217606682baff1ef90591fd48801"} Nov 02 08:10:40.519970 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:10:40.519936 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:10:41.050956 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:41.050921 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" event={"ID":"0dd118ab-44e1-4bd2-95d7-157d2564c23c","Type":"ContainerStarted","Data":"b9c4601f24f259bf7783bb71d71dc1819006bd7dc9e4b23d62c983f215453c5e"} Nov 02 08:10:41.051431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:41.051078 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:41.126604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:41.126569 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:41.274096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:41.273984 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" podStartSLOduration=2.432157185 podStartE2EDuration="7.273968963s" totalImagesPullingTime="4.360656304s" totalInitContainerRuntime="481.155474ms" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:41.227373099 +0000 UTC m=+239.780877327" watchObservedRunningTime="2026-11-02 08:10:41.273968963 +0000 UTC m=+239.827473160" Nov 02 08:10:44.641062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:44.641022 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:44.830337 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:44.830299 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:45.127094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:45.127067 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:45.930292 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:45.930254 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:48.580798 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.580764 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-clxzt"] Nov 02 08:10:48.585119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.585102 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-clxzt" Nov 02 08:10:48.663055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.663024 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbdn\" (UniqueName: \"kubernetes.io/projected/40d72bb6-5223-4b63-ae28-d211b42e70c0-kube-api-access-mhbdn\") pod \"cert-manager-597b95cb84-clxzt\" (UID: \"40d72bb6-5223-4b63-ae28-d211b42e70c0\") " pod="cert-manager/cert-manager-597b95cb84-clxzt" Nov 02 08:10:48.663209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.663063 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40d72bb6-5223-4b63-ae28-d211b42e70c0-bound-sa-token\") pod \"cert-manager-597b95cb84-clxzt\" (UID: \"40d72bb6-5223-4b63-ae28-d211b42e70c0\") " pod="cert-manager/cert-manager-597b95cb84-clxzt" Nov 02 08:10:48.680856 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.680832 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xk886\"" Nov 02 08:10:48.682367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.682348 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-clxzt"] Nov 02 08:10:48.763461 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.763430 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbdn\" (UniqueName: \"kubernetes.io/projected/40d72bb6-5223-4b63-ae28-d211b42e70c0-kube-api-access-mhbdn\") pod \"cert-manager-597b95cb84-clxzt\" (UID: \"40d72bb6-5223-4b63-ae28-d211b42e70c0\") " pod="cert-manager/cert-manager-597b95cb84-clxzt" Nov 02 08:10:48.763610 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.763471 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40d72bb6-5223-4b63-ae28-d211b42e70c0-bound-sa-token\") pod \"cert-manager-597b95cb84-clxzt\" (UID: \"40d72bb6-5223-4b63-ae28-d211b42e70c0\") " pod="cert-manager/cert-manager-597b95cb84-clxzt" Nov 02 08:10:48.819044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.819003 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbdn\" (UniqueName: \"kubernetes.io/projected/40d72bb6-5223-4b63-ae28-d211b42e70c0-kube-api-access-mhbdn\") pod \"cert-manager-597b95cb84-clxzt\" (UID: \"40d72bb6-5223-4b63-ae28-d211b42e70c0\") " pod="cert-manager/cert-manager-597b95cb84-clxzt" Nov 02 08:10:48.820218 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.820198 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40d72bb6-5223-4b63-ae28-d211b42e70c0-bound-sa-token\") pod \"cert-manager-597b95cb84-clxzt\" (UID: \"40d72bb6-5223-4b63-ae28-d211b42e70c0\") " pod="cert-manager/cert-manager-597b95cb84-clxzt" Nov 02 08:10:48.895448 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:48.895354 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-clxzt" Nov 02 08:10:49.259626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:49.259594 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-clxzt"] Nov 02 08:10:49.262910 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:10:49.262884 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d72bb6_5223_4b63_ae28_d211b42e70c0.slice/crio-10b95d01247f05be4835efce47e9deb584d98e7730bc7e7026fc33a1cda4650a WatchSource:0}: Error finding container 10b95d01247f05be4835efce47e9deb584d98e7730bc7e7026fc33a1cda4650a: Status 404 returned error can't find the container with id 10b95d01247f05be4835efce47e9deb584d98e7730bc7e7026fc33a1cda4650a Nov 02 08:10:49.395097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:49.395066 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:49.926624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:49.926585 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:50.097244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:50.097207 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-clxzt" event={"ID":"40d72bb6-5223-4b63-ae28-d211b42e70c0","Type":"ContainerStarted","Data":"0f1d2baa73360733c3fb077b90efe316eb919183d6d890173e8c9bfe9a1ba392"} Nov 02 08:10:50.097244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:50.097249 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-clxzt" event={"ID":"40d72bb6-5223-4b63-ae28-d211b42e70c0","Type":"ContainerStarted","Data":"10b95d01247f05be4835efce47e9deb584d98e7730bc7e7026fc33a1cda4650a"} Nov 02 08:10:50.203719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:50.203605 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-clxzt" podStartSLOduration=2.203587577 podStartE2EDuration="2.203587577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:50.203318656 +0000 UTC m=+248.756822853" watchObservedRunningTime="2026-11-02 08:10:50.203587577 +0000 UTC m=+248.757091777" Nov 02 08:10:50.346154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:50.346108 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:50.500322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:50.500233 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:52.057920 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:52.057889 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b679c64f6-l58kw" Nov 02 08:10:52.339584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:52.339496 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:53.370845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.370809 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7"] Nov 02 08:10:53.373998 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.373981 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" Nov 02 08:10:53.394938 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.394913 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d373af19-25aa-4158-9847-cf62c7db6257-tmp\") pod \"jobset-operator-674cd59dc5-45sf7\" (UID: \"d373af19-25aa-4158-9847-cf62c7db6257\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" Nov 02 08:10:53.395076 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.394955 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jktl\" (UniqueName: \"kubernetes.io/projected/d373af19-25aa-4158-9847-cf62c7db6257-kube-api-access-5jktl\") pod \"jobset-operator-674cd59dc5-45sf7\" (UID: \"d373af19-25aa-4158-9847-cf62c7db6257\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" Nov 02 08:10:53.422724 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.422696 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 08:10:53.468946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.468909 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 08:10:53.469094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.468954 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-htkcj\"" Nov 02 08:10:53.470515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.470495 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7"] Nov 02 08:10:53.496407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.496378 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jktl\" (UniqueName: \"kubernetes.io/projected/d373af19-25aa-4158-9847-cf62c7db6257-kube-api-access-5jktl\") pod \"jobset-operator-674cd59dc5-45sf7\" (UID: \"d373af19-25aa-4158-9847-cf62c7db6257\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" Nov 02 08:10:53.496533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.496437 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d373af19-25aa-4158-9847-cf62c7db6257-tmp\") pod \"jobset-operator-674cd59dc5-45sf7\" (UID: \"d373af19-25aa-4158-9847-cf62c7db6257\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" Nov 02 08:10:53.496784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.496764 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d373af19-25aa-4158-9847-cf62c7db6257-tmp\") pod \"jobset-operator-674cd59dc5-45sf7\" (UID: \"d373af19-25aa-4158-9847-cf62c7db6257\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" Nov 02 08:10:53.549747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.549713 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jktl\" (UniqueName: \"kubernetes.io/projected/d373af19-25aa-4158-9847-cf62c7db6257-kube-api-access-5jktl\") pod \"jobset-operator-674cd59dc5-45sf7\" (UID: \"d373af19-25aa-4158-9847-cf62c7db6257\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" Nov 02 08:10:53.685649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.685604 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" Nov 02 08:10:53.742361 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:53.742302 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:54.050832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:54.050781 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7"] Nov 02 08:10:54.053010 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:10:54.052986 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd373af19_25aa_4158_9847_cf62c7db6257.slice/crio-18928f785c904ed1406ba65055545533fb860fd8856d3f015a265f038f9d0b3a WatchSource:0}: Error finding container 18928f785c904ed1406ba65055545533fb860fd8856d3f015a265f038f9d0b3a: Status 404 returned error can't find the container with id 18928f785c904ed1406ba65055545533fb860fd8856d3f015a265f038f9d0b3a Nov 02 08:10:54.115351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:54.115316 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" event={"ID":"d373af19-25aa-4158-9847-cf62c7db6257","Type":"ContainerStarted","Data":"18928f785c904ed1406ba65055545533fb860fd8856d3f015a265f038f9d0b3a"} Nov 02 08:10:54.500558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:54.500516 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:54.939759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:54.939724 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:55.319417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:55.319336 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:57.131774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:57.131744 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" event={"ID":"d373af19-25aa-4158-9847-cf62c7db6257","Type":"ContainerStarted","Data":"27d2322d648bef03615ed65dcacfada7d6cdc5b4cbf6e72c15ff39f1d0887c08"} Nov 02 08:10:57.277478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:57.277404 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-45sf7" podStartSLOduration=1.260365291 podStartE2EDuration="4.277387405s" totalImagesPullingTime="3.017022114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:57.276923946 +0000 UTC m=+255.830428142" watchObservedRunningTime="2026-11-02 08:10:57.277387405 +0000 UTC m=+255.830891603" Nov 02 08:10:57.533138 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:57.533106 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:59.188213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:59.188159 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:59.559204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:10:59.559092 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:03.933859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:03.933818 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:04.142530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:04.142486 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:09.173555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:09.173514 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:09.435207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:09.435093 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:09.913655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:09.913625 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:10.067468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:10.067436 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:10.519746 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:11:10.519701 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:11:14.340228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:14.340169 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:14.527482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:14.527445 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:15.827365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:15.827324 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:17.421710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.421672 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75"] Nov 02 08:11:17.427375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.427355 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.431807 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.431778 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:17.483274 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.483239 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d2e8ac17-bdcf-4895-acd8-105b444731fc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lwl75\" (UID: \"d2e8ac17-bdcf-4895-acd8-105b444731fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.483444 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.483370 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgqt\" (UniqueName: \"kubernetes.io/projected/d2e8ac17-bdcf-4895-acd8-105b444731fc-kube-api-access-ktgqt\") pod \"kuadrant-console-plugin-66c4fb575d-lwl75\" (UID: \"d2e8ac17-bdcf-4895-acd8-105b444731fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.483518 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.483501 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2e8ac17-bdcf-4895-acd8-105b444731fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lwl75\" (UID: \"d2e8ac17-bdcf-4895-acd8-105b444731fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.522985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.522944 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 08:11:17.522985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.522944 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 08:11:17.523240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.522994 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 08:11:17.523240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.523026 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p96n8\"" Nov 02 08:11:17.523240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.523195 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 08:11:17.524662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.524634 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75"] Nov 02 08:11:17.584383 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.584350 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d2e8ac17-bdcf-4895-acd8-105b444731fc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lwl75\" (UID: \"d2e8ac17-bdcf-4895-acd8-105b444731fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.584579 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.584398 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgqt\" (UniqueName: \"kubernetes.io/projected/d2e8ac17-bdcf-4895-acd8-105b444731fc-kube-api-access-ktgqt\") pod \"kuadrant-console-plugin-66c4fb575d-lwl75\" (UID: \"d2e8ac17-bdcf-4895-acd8-105b444731fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.584579 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.584443 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2e8ac17-bdcf-4895-acd8-105b444731fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lwl75\" (UID: \"d2e8ac17-bdcf-4895-acd8-105b444731fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.585061 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.585037 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d2e8ac17-bdcf-4895-acd8-105b444731fc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lwl75\" (UID: \"d2e8ac17-bdcf-4895-acd8-105b444731fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.587067 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.587043 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2e8ac17-bdcf-4895-acd8-105b444731fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lwl75\" (UID: \"d2e8ac17-bdcf-4895-acd8-105b444731fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.633856 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.633824 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:17.638030 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.638001 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgqt\" (UniqueName: \"kubernetes.io/projected/d2e8ac17-bdcf-4895-acd8-105b444731fc-kube-api-access-ktgqt\") pod \"kuadrant-console-plugin-66c4fb575d-lwl75\" (UID: \"d2e8ac17-bdcf-4895-acd8-105b444731fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:17.737982 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:17.737896 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" Nov 02 08:11:18.101044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:18.100963 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75"] Nov 02 08:11:18.103433 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:11:18.103406 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2e8ac17_bdcf_4895_acd8_105b444731fc.slice/crio-743eae7cf86296eaadfcba907e35250476e92fdac06f4dd72f8ecd3fd92d79a2 WatchSource:0}: Error finding container 743eae7cf86296eaadfcba907e35250476e92fdac06f4dd72f8ecd3fd92d79a2: Status 404 returned error can't find the container with id 743eae7cf86296eaadfcba907e35250476e92fdac06f4dd72f8ecd3fd92d79a2 Nov 02 08:11:18.231543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:18.231505 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" event={"ID":"d2e8ac17-bdcf-4895-acd8-105b444731fc","Type":"ContainerStarted","Data":"743eae7cf86296eaadfcba907e35250476e92fdac06f4dd72f8ecd3fd92d79a2"} Nov 02 08:11:18.776965 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:18.776933 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:18.942514 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:18.942476 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:19.324192 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:19.324144 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:19.491071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:19.491040 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:19.720395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:19.720355 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:19.878112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:19.878077 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:24.072100 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:24.072056 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:24.275873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:24.275776 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" event={"ID":"d2e8ac17-bdcf-4895-acd8-105b444731fc","Type":"ContainerStarted","Data":"fd6cfadbb7a202d4e2bc92a8c0ac178f43719a24fff97b69dd5cce0c9ff5e4e0"} Nov 02 08:11:24.288914 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:24.288886 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:24.381787 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:24.381731 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lwl75" podStartSLOduration=1.5160314499999998 podStartE2EDuration="7.38171894s" totalImagesPullingTime="5.86568749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:11:24.381222714 +0000 UTC m=+282.934726909" watchObservedRunningTime="2026-11-02 08:11:24.38171894 +0000 UTC m=+282.935223137" Nov 02 08:11:24.572132 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:24.571820 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:26.189309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.189262 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:26.205767 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.205738 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2"] Nov 02 08:11:26.209325 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.209300 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.261067 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.261037 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 08:11:26.273351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.273330 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 08:11:26.308576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.308542 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2"] Nov 02 08:11:26.355881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.355850 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 08:11:26.355881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.355874 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 08:11:26.355881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.355860 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qkpvv\"" Nov 02 08:11:26.363245 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.363219 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9czh\" (UniqueName: \"kubernetes.io/projected/5bfaa88b-36fe-4db7-8638-a30f7a0b7b87-kube-api-access-j9czh\") pod \"kube-auth-proxy-7ddcd68ff8-ckxc2\" (UID: \"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87\") " pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.363376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.363283 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfaa88b-36fe-4db7-8638-a30f7a0b7b87-tls-certs\") pod \"kube-auth-proxy-7ddcd68ff8-ckxc2\" (UID: \"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87\") " pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.363417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.363389 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa88b-36fe-4db7-8638-a30f7a0b7b87-tmp\") pod \"kube-auth-proxy-7ddcd68ff8-ckxc2\" (UID: \"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87\") " pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.405480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.405442 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:26.464232 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.464122 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfaa88b-36fe-4db7-8638-a30f7a0b7b87-tls-certs\") pod \"kube-auth-proxy-7ddcd68ff8-ckxc2\" (UID: \"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87\") " pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.464232 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.464198 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa88b-36fe-4db7-8638-a30f7a0b7b87-tmp\") pod \"kube-auth-proxy-7ddcd68ff8-ckxc2\" (UID: \"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87\") " pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.464474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.464246 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9czh\" (UniqueName: \"kubernetes.io/projected/5bfaa88b-36fe-4db7-8638-a30f7a0b7b87-kube-api-access-j9czh\") pod \"kube-auth-proxy-7ddcd68ff8-ckxc2\" (UID: \"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87\") " pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.467194 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.467156 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfaa88b-36fe-4db7-8638-a30f7a0b7b87-tls-certs\") pod \"kube-auth-proxy-7ddcd68ff8-ckxc2\" (UID: \"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87\") " pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.468323 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.468295 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa88b-36fe-4db7-8638-a30f7a0b7b87-tmp\") pod \"kube-auth-proxy-7ddcd68ff8-ckxc2\" (UID: \"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87\") " pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.523067 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.523033 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9czh\" (UniqueName: \"kubernetes.io/projected/5bfaa88b-36fe-4db7-8638-a30f7a0b7b87-kube-api-access-j9czh\") pod \"kube-auth-proxy-7ddcd68ff8-ckxc2\" (UID: \"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87\") " pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.578710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.578674 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:26.821080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.820994 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" Nov 02 08:11:26.849960 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:26.849923 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:27.029334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:27.029299 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:27.197285 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:27.197260 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2"] Nov 02 08:11:27.199699 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:11:27.199669 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfaa88b_36fe_4db7_8638_a30f7a0b7b87.slice/crio-0356b34a39a35ecc7f0983d7ea6c5cd4ff58870c19eb58c766fb3d4d7849c052 WatchSource:0}: Error finding container 0356b34a39a35ecc7f0983d7ea6c5cd4ff58870c19eb58c766fb3d4d7849c052: Status 404 returned error can't find the container with id 0356b34a39a35ecc7f0983d7ea6c5cd4ff58870c19eb58c766fb3d4d7849c052 Nov 02 08:11:27.234950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:27.234917 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:27.292571 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:27.292522 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" event={"ID":"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87","Type":"ContainerStarted","Data":"0356b34a39a35ecc7f0983d7ea6c5cd4ff58870c19eb58c766fb3d4d7849c052"} Nov 02 08:11:28.376816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:28.376781 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:28.584522 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:28.584487 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:28.824473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:28.824433 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:29.179766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.179673 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:29.326990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.326950 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:29.651487 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.651416 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q"] Nov 02 08:11:29.657093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.657071 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.752734 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.752686 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 08:11:29.752919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.752755 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-lpttm\"" Nov 02 08:11:29.752919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.752811 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 08:11:29.753027 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.752938 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 08:11:29.754525 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.754498 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q"] Nov 02 08:11:29.795193 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.795149 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhzq\" (UniqueName: \"kubernetes.io/projected/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-kube-api-access-bmhzq\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.795375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.795279 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-cert\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.795375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.795341 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-metrics-certs\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.795491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.795377 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-manager-config\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.896637 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.896603 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhzq\" (UniqueName: \"kubernetes.io/projected/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-kube-api-access-bmhzq\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.896838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.896650 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-cert\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.896838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.896709 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-metrics-certs\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.896838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.896742 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-manager-config\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.897501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.897476 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-manager-config\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.899588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.899566 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-cert\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.899723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.899654 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-metrics-certs\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.949798 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.949768 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhzq\" (UniqueName: \"kubernetes.io/projected/65f5bc98-5c00-4ce1-a4ad-6bac42d3810d-kube-api-access-bmhzq\") pod \"jobset-controller-manager-5866c976c5-ksz8q\" (UID: \"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d\") " pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:29.970590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:29.970561 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:30.370638 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:30.370553 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q"] Nov 02 08:11:30.374093 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:11:30.374064 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f5bc98_5c00_4ce1_a4ad_6bac42d3810d.slice/crio-5cd84d1db6783ef9e3515a69407f4c98d0e1b94ccab2b1e6e5dab3ca39058819 WatchSource:0}: Error finding container 5cd84d1db6783ef9e3515a69407f4c98d0e1b94ccab2b1e6e5dab3ca39058819: Status 404 returned error can't find the container with id 5cd84d1db6783ef9e3515a69407f4c98d0e1b94ccab2b1e6e5dab3ca39058819 Nov 02 08:11:30.531234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:30.531201 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:30.708308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:30.708275 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:31.314047 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:31.314006 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" event={"ID":"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d","Type":"ContainerStarted","Data":"5cd84d1db6783ef9e3515a69407f4c98d0e1b94ccab2b1e6e5dab3ca39058819"} Nov 02 08:11:31.315775 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:31.315741 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" event={"ID":"5bfaa88b-36fe-4db7-8638-a30f7a0b7b87","Type":"ContainerStarted","Data":"d40780de7fe99e5b653deae3b190cb58dde0b09958cf49b6e1c923e54757b8f7"} Nov 02 08:11:31.421205 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:31.421142 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ddcd68ff8-ckxc2" podStartSLOduration=2.083336355 podStartE2EDuration="5.421128173s" totalImagesPullingTime="3.337791818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:11:31.420666703 +0000 UTC m=+289.974170899" watchObservedRunningTime="2026-11-02 08:11:31.421128173 +0000 UTC m=+289.974632369" Nov 02 08:11:33.617396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:33.617358 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:33.904863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:33.904757 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:34.062351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:34.062314 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:34.922062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:34.922021 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:35.134834 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:35.134734 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:35.337721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:35.337685 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" event={"ID":"65f5bc98-5c00-4ce1-a4ad-6bac42d3810d","Type":"ContainerStarted","Data":"4f7a06b6e0977697fd8074b5cb8deb935ddb15ccd959f54c80f59223026629fc"} Nov 02 08:11:35.337919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:35.337808 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:35.445788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:35.445740 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" podStartSLOduration=2.09323851 podStartE2EDuration="6.445724099s" totalImagesPullingTime="4.352485589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:11:35.445569471 +0000 UTC m=+293.999073666" watchObservedRunningTime="2026-11-02 08:11:35.445724099 +0000 UTC m=+293.999228295" Nov 02 08:11:38.711676 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:38.711636 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:38.910804 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:38.910768 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:40.520257 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:11:40.520213 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:11:41.595406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:41.595374 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:11:41.596656 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:41.596629 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:11:42.760443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:42.760408 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:42.919304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:42.919271 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:46.347584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:46.347558 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5866c976c5-ksz8q" Nov 02 08:11:49.701686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:49.701644 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:56.528925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:56.528892 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:56.755063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:56.755027 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:57.280997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:57.280957 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:59.450225 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:11:59.450170 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:10.521248 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:12:10.521211 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:12:29.512304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:29.512256 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:29.666533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:29.666497 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:29.818136 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:29.818043 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:30.019808 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:30.019770 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:33.138740 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:33.138704 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:33.291853 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:33.291821 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:36.913674 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:36.913640 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:37.183039 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:37.182949 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:40.519551 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:12:40.519516 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:12:41.945681 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:41.945644 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:42.154044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:42.153998 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:43.802396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:43.802362 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:43.959948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:43.959913 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:44.113063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.112985 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:44.256384 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.256349 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:44.414987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.414960 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:44.569587 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.569552 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:44.743412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.743323 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:44.858754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.858714 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fbqk9"] Nov 02 08:12:44.862396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.862376 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:44.900365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.900339 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:44.910871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.910848 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 08:12:44.926485 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.926456 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqc7\" (UniqueName: \"kubernetes.io/projected/1fd6f36b-0fce-4b8a-ae80-f163a73f87c9-kube-api-access-cvqc7\") pod \"model-serving-api-75bd89fdfc-fbqk9\" (UID: \"1fd6f36b-0fce-4b8a-ae80-f163a73f87c9\") " pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:44.926632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.926573 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd6f36b-0fce-4b8a-ae80-f163a73f87c9-tls-certs\") pod \"model-serving-api-75bd89fdfc-fbqk9\" (UID: \"1fd6f36b-0fce-4b8a-ae80-f163a73f87c9\") " pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:44.957046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.957016 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 08:12:44.957169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.957048 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mv2b9\"" Nov 02 08:12:44.957169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.957092 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 08:12:44.958677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:44.958657 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fbqk9"] Nov 02 08:12:45.027201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:45.027100 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd6f36b-0fce-4b8a-ae80-f163a73f87c9-tls-certs\") pod \"model-serving-api-75bd89fdfc-fbqk9\" (UID: \"1fd6f36b-0fce-4b8a-ae80-f163a73f87c9\") " pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:45.027201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:45.027152 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqc7\" (UniqueName: \"kubernetes.io/projected/1fd6f36b-0fce-4b8a-ae80-f163a73f87c9-kube-api-access-cvqc7\") pod \"model-serving-api-75bd89fdfc-fbqk9\" (UID: \"1fd6f36b-0fce-4b8a-ae80-f163a73f87c9\") " pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:45.029706 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:45.029679 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd6f36b-0fce-4b8a-ae80-f163a73f87c9-tls-certs\") pod \"model-serving-api-75bd89fdfc-fbqk9\" (UID: \"1fd6f36b-0fce-4b8a-ae80-f163a73f87c9\") " pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:45.080394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:45.080362 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqc7\" (UniqueName: \"kubernetes.io/projected/1fd6f36b-0fce-4b8a-ae80-f163a73f87c9-kube-api-access-cvqc7\") pod \"model-serving-api-75bd89fdfc-fbqk9\" (UID: \"1fd6f36b-0fce-4b8a-ae80-f163a73f87c9\") " pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:45.175065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:45.175021 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:45.537885 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:45.537851 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fbqk9"] Nov 02 08:12:45.541027 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:12:45.540991 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd6f36b_0fce_4b8a_ae80_f163a73f87c9.slice/crio-ccc5813322a0c52dac3de37e31350dd7228cab0cce6d144be8255c7164575d74 WatchSource:0}: Error finding container ccc5813322a0c52dac3de37e31350dd7228cab0cce6d144be8255c7164575d74: Status 404 returned error can't find the container with id ccc5813322a0c52dac3de37e31350dd7228cab0cce6d144be8255c7164575d74 Nov 02 08:12:45.543256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:45.543241 6160 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:12:45.673535 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:45.673493 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" event={"ID":"1fd6f36b-0fce-4b8a-ae80-f163a73f87c9","Type":"ContainerStarted","Data":"ccc5813322a0c52dac3de37e31350dd7228cab0cce6d144be8255c7164575d74"} Nov 02 08:12:46.006728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:46.006681 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:48.689938 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:48.689899 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" event={"ID":"1fd6f36b-0fce-4b8a-ae80-f163a73f87c9","Type":"ContainerStarted","Data":"11d91011cfa0fc2aa10a7c4e69bc17159bf6cc90bb3d6bdf68052b3c3f047e3b"} Nov 02 08:12:48.690355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:48.690036 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:48.796804 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:48.796750 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" podStartSLOduration=2.2813931849999998 podStartE2EDuration="4.796734204s" totalImagesPullingTime="2.515341019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:48.79455757 +0000 UTC m=+367.348061791" watchObservedRunningTime="2026-11-02 08:12:48.796734204 +0000 UTC m=+367.350238484" Nov 02 08:12:50.963456 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:50.963382 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:51.172984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:51.172945 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:51.318628 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:51.318543 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:51.520380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:51.520337 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:51.679273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:51.679162 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:51.841987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:51.841948 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:52.000474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:52.000388 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:52.151651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:52.151616 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:59.698977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:59.698943 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-fbqk9" Nov 02 08:12:59.789459 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:12:59.789423 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:10.520348 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:13:10.520306 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:13:10.687903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:10.687858 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:10.897614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:10.897520 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:12.825701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:12.825668 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:13.041401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.041364 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:13.202449 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.202412 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:13.353967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.353927 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:13.859162 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.859125 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97955854-2c5bd"] Nov 02 08:13:13.863846 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.863828 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.886297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.886263 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-maas-api-tls\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.886297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.886302 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hq6h\" (UniqueName: \"kubernetes.io/projected/3297628c-8815-4af6-ba07-7c2f19e1ac38-kube-api-access-8hq6h\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.886531 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.886432 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3297628c-8815-4af6-ba07-7c2f19e1ac38-openshift-service-ca\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.886531 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.886475 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-metrics-certs\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.961383 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.961344 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n86zq\"" Nov 02 08:13:13.961585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.961391 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 08:13:13.961585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.961443 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:13:13.961585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.961484 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:13:13.961585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.961518 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 08:13:13.963157 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.963137 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97955854-2c5bd"] Nov 02 08:13:13.986953 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.986926 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3297628c-8815-4af6-ba07-7c2f19e1ac38-openshift-service-ca\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.987072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.986968 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-metrics-certs\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.987072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.986992 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-maas-api-tls\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.987072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.987016 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hq6h\" (UniqueName: \"kubernetes.io/projected/3297628c-8815-4af6-ba07-7c2f19e1ac38-kube-api-access-8hq6h\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.987618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.987595 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3297628c-8815-4af6-ba07-7c2f19e1ac38-openshift-service-ca\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.989570 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.989549 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-maas-api-tls\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:13.989671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:13.989597 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-metrics-certs\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:14.040923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:14.040893 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hq6h\" (UniqueName: \"kubernetes.io/projected/3297628c-8815-4af6-ba07-7c2f19e1ac38-kube-api-access-8hq6h\") pod \"maas-api-7b97955854-2c5bd\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:14.176724 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:14.176693 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:14.542364 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:14.542169 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97955854-2c5bd"] Nov 02 08:13:14.545100 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:13:14.545074 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3297628c_8815_4af6_ba07_7c2f19e1ac38.slice/crio-a31899c8eec7c407fed4825c24f0dd466b3f54582c74306b2a60d68973092f9a WatchSource:0}: Error finding container a31899c8eec7c407fed4825c24f0dd466b3f54582c74306b2a60d68973092f9a: Status 404 returned error can't find the container with id a31899c8eec7c407fed4825c24f0dd466b3f54582c74306b2a60d68973092f9a Nov 02 08:13:14.808797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:14.808706 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" event={"ID":"3297628c-8815-4af6-ba07-7c2f19e1ac38","Type":"ContainerStarted","Data":"a31899c8eec7c407fed4825c24f0dd466b3f54582c74306b2a60d68973092f9a"} Nov 02 08:13:20.838882 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:20.838791 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" event={"ID":"3297628c-8815-4af6-ba07-7c2f19e1ac38","Type":"ContainerStarted","Data":"17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd"} Nov 02 08:13:20.839346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:20.838931 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:20.946557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:20.946505 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" podStartSLOduration=2.01427827 podStartE2EDuration="7.946490286s" totalImagesPullingTime="5.932212016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:20.94444148 +0000 UTC m=+399.497945698" watchObservedRunningTime="2026-11-02 08:13:20.946490286 +0000 UTC m=+399.499994494" Nov 02 08:13:26.848803 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:26.848773 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:27.356044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:27.355993 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:27.665709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:27.665673 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:37.186794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.186760 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97955854-2c5bd"] Nov 02 08:13:37.187382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.186954 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" podUID="3297628c-8815-4af6-ba07-7c2f19e1ac38" containerName="maas-api" containerID="cri-o://17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd" gracePeriod=30 Nov 02 08:13:37.421005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.420977 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:37.486050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.485972 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-maas-api-tls\") pod \"3297628c-8815-4af6-ba07-7c2f19e1ac38\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " Nov 02 08:13:37.486050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.486021 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-metrics-certs\") pod \"3297628c-8815-4af6-ba07-7c2f19e1ac38\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " Nov 02 08:13:37.486050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.486044 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3297628c-8815-4af6-ba07-7c2f19e1ac38-kube-api-access-8hq6h\" (UniqueName: \"kubernetes.io/projected/3297628c-8815-4af6-ba07-7c2f19e1ac38-kube-api-access-8hq6h\") pod \"3297628c-8815-4af6-ba07-7c2f19e1ac38\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " Nov 02 08:13:37.486318 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.486111 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3297628c-8815-4af6-ba07-7c2f19e1ac38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3297628c-8815-4af6-ba07-7c2f19e1ac38-openshift-service-ca\") pod \"3297628c-8815-4af6-ba07-7c2f19e1ac38\" (UID: \"3297628c-8815-4af6-ba07-7c2f19e1ac38\") " Nov 02 08:13:37.486569 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.486537 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3297628c-8815-4af6-ba07-7c2f19e1ac38-openshift-service-ca" pod "3297628c-8815-4af6-ba07-7c2f19e1ac38" (UID: "3297628c-8815-4af6-ba07-7c2f19e1ac38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:13:37.488544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.488521 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-maas-api-tls" pod "3297628c-8815-4af6-ba07-7c2f19e1ac38" (UID: "3297628c-8815-4af6-ba07-7c2f19e1ac38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:13:37.488633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.488577 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-metrics-certs" pod "3297628c-8815-4af6-ba07-7c2f19e1ac38" (UID: "3297628c-8815-4af6-ba07-7c2f19e1ac38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:13:37.488675 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.488628 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3297628c-8815-4af6-ba07-7c2f19e1ac38-kube-api-access-8hq6h" pod "3297628c-8815-4af6-ba07-7c2f19e1ac38" (UID: "3297628c-8815-4af6-ba07-7c2f19e1ac38"). InnerVolumeSpecName "kube-api-access-8hq6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:13:37.586941 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.586908 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:13:37.586941 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.586934 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3297628c-8815-4af6-ba07-7c2f19e1ac38-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:13:37.587134 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.586948 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hq6h\" (UniqueName: \"kubernetes.io/projected/3297628c-8815-4af6-ba07-7c2f19e1ac38-kube-api-access-8hq6h\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:13:37.587134 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.586962 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3297628c-8815-4af6-ba07-7c2f19e1ac38-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:13:37.921038 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.921003 6160 generic.go:356] "Generic (PLEG): container finished" podID="3297628c-8815-4af6-ba07-7c2f19e1ac38" containerID="17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd" exitCode=0 Nov 02 08:13:37.921258 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.921053 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" Nov 02 08:13:37.921258 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.921089 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" event={"ID":"3297628c-8815-4af6-ba07-7c2f19e1ac38","Type":"ContainerDied","Data":"17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd"} Nov 02 08:13:37.921258 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.921145 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97955854-2c5bd" event={"ID":"3297628c-8815-4af6-ba07-7c2f19e1ac38","Type":"ContainerDied","Data":"a31899c8eec7c407fed4825c24f0dd466b3f54582c74306b2a60d68973092f9a"} Nov 02 08:13:37.921258 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.921201 6160 scope.go:122] "RemoveContainer" containerID="17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd" Nov 02 08:13:37.930096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.930077 6160 scope.go:122] "RemoveContainer" containerID="17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd" Nov 02 08:13:37.930422 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:13:37.930404 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd\": container with ID starting with 17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd not found: ID does not exist" containerID="17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd" Nov 02 08:13:37.930476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:37.930435 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd"} err="failed to get container status \"17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd\": rpc error: code = NotFound desc = could not find container \"17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd\": container with ID starting with 17a42004aecbad0fe0dd7eb2a9cc8a214ad64ed716a73624b8e22190f2dce5dd not found: ID does not exist" Nov 02 08:13:38.073538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:38.073507 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97955854-2c5bd"] Nov 02 08:13:38.119700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:38.119671 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97955854-2c5bd"] Nov 02 08:13:39.582660 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:39.582621 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3297628c-8815-4af6-ba07-7c2f19e1ac38" path="/var/lib/kubelet/pods/3297628c-8815-4af6-ba07-7c2f19e1ac38/volumes" Nov 02 08:13:40.519078 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:13:40.519037 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:13:55.057925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:13:55.057888 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:00.139155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:00.139117 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:10.518894 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:14:10.518855 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:14:27.363429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.363344 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2kkhq"] Nov 02 08:14:27.363778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.363649 6160 cpu_manager.go:414] "containerMap: removing container" podUID="3297628c-8815-4af6-ba07-7c2f19e1ac38" containerName="maas-api" Nov 02 08:14:27.363778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.363660 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3297628c-8815-4af6-ba07-7c2f19e1ac38" containerName="maas-api" Nov 02 08:14:27.363778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.363718 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="3297628c-8815-4af6-ba07-7c2f19e1ac38" containerName="maas-api" Nov 02 08:14:27.366658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.366641 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:27.388516 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.388488 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:27.400070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.400039 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/391cad15-d20a-4d7c-a118-108d43622ac2-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-2kkhq\" (UID: \"391cad15-d20a-4d7c-a118-108d43622ac2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:27.400215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.400090 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjct\" (UniqueName: \"kubernetes.io/projected/391cad15-d20a-4d7c-a118-108d43622ac2-kube-api-access-xkjct\") pod \"ai-gateway-controller-7d85675f98-2kkhq\" (UID: \"391cad15-d20a-4d7c-a118-108d43622ac2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:27.415412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.415385 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 08:14:27.461686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.461658 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 08:14:27.461686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.461668 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9j8gb\"" Nov 02 08:14:27.463556 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.463534 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2kkhq"] Nov 02 08:14:27.501655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.501607 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjct\" (UniqueName: \"kubernetes.io/projected/391cad15-d20a-4d7c-a118-108d43622ac2-kube-api-access-xkjct\") pod \"ai-gateway-controller-7d85675f98-2kkhq\" (UID: \"391cad15-d20a-4d7c-a118-108d43622ac2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:27.501798 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.501738 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/391cad15-d20a-4d7c-a118-108d43622ac2-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-2kkhq\" (UID: \"391cad15-d20a-4d7c-a118-108d43622ac2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:27.504715 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.504688 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/391cad15-d20a-4d7c-a118-108d43622ac2-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-2kkhq\" (UID: \"391cad15-d20a-4d7c-a118-108d43622ac2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:27.554893 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.554869 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjct\" (UniqueName: \"kubernetes.io/projected/391cad15-d20a-4d7c-a118-108d43622ac2-kube-api-access-xkjct\") pod \"ai-gateway-controller-7d85675f98-2kkhq\" (UID: \"391cad15-d20a-4d7c-a118-108d43622ac2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:27.597086 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.597050 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:27.678627 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:27.678597 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:28.040870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:28.040842 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2kkhq"] Nov 02 08:14:28.042892 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:14:28.042864 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391cad15_d20a_4d7c_a118_108d43622ac2.slice/crio-77ed19c8bfe393c2f5145c7fb80159a94953de1cc288b5948f550eeb283f5b9e WatchSource:0}: Error finding container 77ed19c8bfe393c2f5145c7fb80159a94953de1cc288b5948f550eeb283f5b9e: Status 404 returned error can't find the container with id 77ed19c8bfe393c2f5145c7fb80159a94953de1cc288b5948f550eeb283f5b9e Nov 02 08:14:28.153163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:28.153130 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" event={"ID":"391cad15-d20a-4d7c-a118-108d43622ac2","Type":"ContainerStarted","Data":"77ed19c8bfe393c2f5145c7fb80159a94953de1cc288b5948f550eeb283f5b9e"} Nov 02 08:14:32.173936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:32.173901 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" event={"ID":"391cad15-d20a-4d7c-a118-108d43622ac2","Type":"ContainerStarted","Data":"0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789"} Nov 02 08:14:32.174346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:32.173962 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:32.282582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:32.282524 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" podStartSLOduration=2.165289695 podStartE2EDuration="5.282505047s" totalImagesPullingTime="3.117215352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:14:32.280227682 +0000 UTC m=+470.833731876" watchObservedRunningTime="2026-11-02 08:14:32.282505047 +0000 UTC m=+470.836009243" Nov 02 08:14:34.984199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:34.984143 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:35.207014 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.206981 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:35.378164 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.375933 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j82zb"] Nov 02 08:14:35.380565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.380530 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:35.431426 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.431402 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 08:14:35.470162 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.470126 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2087bcd1-2ad8-4ce0-a356-99720eb3c494-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j82zb\" (UID: \"2087bcd1-2ad8-4ce0-a356-99720eb3c494\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:35.470348 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.470310 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klx52\" (UniqueName: \"kubernetes.io/projected/2087bcd1-2ad8-4ce0-a356-99720eb3c494-kube-api-access-klx52\") pod \"payload-pre-processing-7b98bb678c-j82zb\" (UID: \"2087bcd1-2ad8-4ce0-a356-99720eb3c494\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:35.477280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.477258 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-85f2k\"" Nov 02 08:14:35.478959 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.478937 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j82zb"] Nov 02 08:14:35.530020 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.529979 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-4h2lv"] Nov 02 08:14:35.533417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.533395 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:35.570891 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.570865 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2087bcd1-2ad8-4ce0-a356-99720eb3c494-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j82zb\" (UID: \"2087bcd1-2ad8-4ce0-a356-99720eb3c494\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:35.571030 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.570901 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7297abc-ee50-4bd9-8065-99c7cd95357f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-4h2lv\" (UID: \"c7297abc-ee50-4bd9-8065-99c7cd95357f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:35.571080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.571045 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lzgq\" (UniqueName: \"kubernetes.io/projected/c7297abc-ee50-4bd9-8065-99c7cd95357f-kube-api-access-5lzgq\") pod \"payload-processing-55fbbbf9dc-4h2lv\" (UID: \"c7297abc-ee50-4bd9-8065-99c7cd95357f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:35.571231 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.571213 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klx52\" (UniqueName: \"kubernetes.io/projected/2087bcd1-2ad8-4ce0-a356-99720eb3c494-kube-api-access-klx52\") pod \"payload-pre-processing-7b98bb678c-j82zb\" (UID: \"2087bcd1-2ad8-4ce0-a356-99720eb3c494\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:35.571570 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.571554 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2087bcd1-2ad8-4ce0-a356-99720eb3c494-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j82zb\" (UID: \"2087bcd1-2ad8-4ce0-a356-99720eb3c494\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:35.587025 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.587000 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-4h2lv"] Nov 02 08:14:35.624411 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.624389 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klx52\" (UniqueName: \"kubernetes.io/projected/2087bcd1-2ad8-4ce0-a356-99720eb3c494-kube-api-access-klx52\") pod \"payload-pre-processing-7b98bb678c-j82zb\" (UID: \"2087bcd1-2ad8-4ce0-a356-99720eb3c494\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:35.672077 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.672048 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lzgq\" (UniqueName: \"kubernetes.io/projected/c7297abc-ee50-4bd9-8065-99c7cd95357f-kube-api-access-5lzgq\") pod \"payload-processing-55fbbbf9dc-4h2lv\" (UID: \"c7297abc-ee50-4bd9-8065-99c7cd95357f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:35.672345 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.672324 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7297abc-ee50-4bd9-8065-99c7cd95357f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-4h2lv\" (UID: \"c7297abc-ee50-4bd9-8065-99c7cd95357f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:35.672950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.672929 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7297abc-ee50-4bd9-8065-99c7cd95357f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-4h2lv\" (UID: \"c7297abc-ee50-4bd9-8065-99c7cd95357f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:35.693019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.692994 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:35.725277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.725250 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lzgq\" (UniqueName: \"kubernetes.io/projected/c7297abc-ee50-4bd9-8065-99c7cd95357f-kube-api-access-5lzgq\") pod \"payload-processing-55fbbbf9dc-4h2lv\" (UID: \"c7297abc-ee50-4bd9-8065-99c7cd95357f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:35.846055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:35.846026 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:36.059250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:36.059227 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j82zb"] Nov 02 08:14:36.060954 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:14:36.060916 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087bcd1_2ad8_4ce0_a356_99720eb3c494.slice/crio-3eec2514997a81f0b04fc1bfe9ebfc375ce84421d5e68936f4750dfe32a2b359 WatchSource:0}: Error finding container 3eec2514997a81f0b04fc1bfe9ebfc375ce84421d5e68936f4750dfe32a2b359: Status 404 returned error can't find the container with id 3eec2514997a81f0b04fc1bfe9ebfc375ce84421d5e68936f4750dfe32a2b359 Nov 02 08:14:36.193833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:36.193799 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" event={"ID":"2087bcd1-2ad8-4ce0-a356-99720eb3c494","Type":"ContainerStarted","Data":"3eec2514997a81f0b04fc1bfe9ebfc375ce84421d5e68936f4750dfe32a2b359"} Nov 02 08:14:36.208943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:36.208922 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-4h2lv"] Nov 02 08:14:36.211603 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:14:36.211576 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7297abc_ee50_4bd9_8065_99c7cd95357f.slice/crio-4101482565a3deb4b9a76fc800fe2aa04e905cbb39cfdc19b68e205fd11ac67b WatchSource:0}: Error finding container 4101482565a3deb4b9a76fc800fe2aa04e905cbb39cfdc19b68e205fd11ac67b: Status 404 returned error can't find the container with id 4101482565a3deb4b9a76fc800fe2aa04e905cbb39cfdc19b68e205fd11ac67b Nov 02 08:14:37.199591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:37.199559 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" event={"ID":"c7297abc-ee50-4bd9-8065-99c7cd95357f","Type":"ContainerStarted","Data":"4101482565a3deb4b9a76fc800fe2aa04e905cbb39cfdc19b68e205fd11ac67b"} Nov 02 08:14:40.218032 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:40.217987 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" event={"ID":"c7297abc-ee50-4bd9-8065-99c7cd95357f","Type":"ContainerStarted","Data":"bfbeb7f144962a9ae8353ed45870f6f1a5338fc1f7b9b8b7ab4ab65596bd35f0"} Nov 02 08:14:40.218573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:40.218144 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:40.219623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:40.219601 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" event={"ID":"2087bcd1-2ad8-4ce0-a356-99720eb3c494","Type":"ContainerStarted","Data":"45f3656c2568a2e6ad3bac84dd4a0c17699141e44019c0e3df8bd2eca8a307dc"} Nov 02 08:14:40.219732 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:40.219721 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:40.324734 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:40.324665 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" podStartSLOduration=2.205722052 podStartE2EDuration="5.324646868s" totalImagesPullingTime="3.118924816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:14:40.323662732 +0000 UTC m=+478.877166928" watchObservedRunningTime="2026-11-02 08:14:40.324646868 +0000 UTC m=+478.878151067" Nov 02 08:14:40.431573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:40.431520 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" podStartSLOduration=2.270516388 podStartE2EDuration="5.43150539s" totalImagesPullingTime="3.160989002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:14:40.429564006 +0000 UTC m=+478.983068229" watchObservedRunningTime="2026-11-02 08:14:40.43150539 +0000 UTC m=+478.985009587" Nov 02 08:14:40.519916 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:14:40.519832 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:14:43.181324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:43.181292 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:14:48.885071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:48.885036 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:49.038843 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:49.038807 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:51.227255 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:51.227223 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-4h2lv" Nov 02 08:14:51.227636 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:51.227287 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j82zb" Nov 02 08:14:53.513450 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.513402 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-c9959c9bb-l695c"] Nov 02 08:14:53.519458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.519434 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:14:53.572315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.572283 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-c9959c9bb-l695c"] Nov 02 08:14:53.631122 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.631088 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwvq\" (UniqueName: \"kubernetes.io/projected/74e96d56-8e33-4351-814b-3d9781be59fd-kube-api-access-tqwvq\") pod \"ai-gateway-controller-c9959c9bb-l695c\" (UID: \"74e96d56-8e33-4351-814b-3d9781be59fd\") " pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:14:53.631122 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.631126 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74e96d56-8e33-4351-814b-3d9781be59fd-metrics-certs\") pod \"ai-gateway-controller-c9959c9bb-l695c\" (UID: \"74e96d56-8e33-4351-814b-3d9781be59fd\") " pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:14:53.732219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.732148 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwvq\" (UniqueName: \"kubernetes.io/projected/74e96d56-8e33-4351-814b-3d9781be59fd-kube-api-access-tqwvq\") pod \"ai-gateway-controller-c9959c9bb-l695c\" (UID: \"74e96d56-8e33-4351-814b-3d9781be59fd\") " pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:14:53.732219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.732230 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74e96d56-8e33-4351-814b-3d9781be59fd-metrics-certs\") pod \"ai-gateway-controller-c9959c9bb-l695c\" (UID: \"74e96d56-8e33-4351-814b-3d9781be59fd\") " pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:14:53.734943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.734910 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74e96d56-8e33-4351-814b-3d9781be59fd-metrics-certs\") pod \"ai-gateway-controller-c9959c9bb-l695c\" (UID: \"74e96d56-8e33-4351-814b-3d9781be59fd\") " pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:14:53.785717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.785645 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwvq\" (UniqueName: \"kubernetes.io/projected/74e96d56-8e33-4351-814b-3d9781be59fd-kube-api-access-tqwvq\") pod \"ai-gateway-controller-c9959c9bb-l695c\" (UID: \"74e96d56-8e33-4351-814b-3d9781be59fd\") " pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:14:53.832913 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:53.832880 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:14:54.201035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:54.200999 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-c9959c9bb-l695c"] Nov 02 08:14:54.203773 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:14:54.203746 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e96d56_8e33_4351_814b_3d9781be59fd.slice/crio-7e50dcc5e9bab2384723f21ee509cf18d10e7d7a10663a59b1ae27b7924102ea WatchSource:0}: Error finding container 7e50dcc5e9bab2384723f21ee509cf18d10e7d7a10663a59b1ae27b7924102ea: Status 404 returned error can't find the container with id 7e50dcc5e9bab2384723f21ee509cf18d10e7d7a10663a59b1ae27b7924102ea Nov 02 08:14:54.288219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:54.288162 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" event={"ID":"74e96d56-8e33-4351-814b-3d9781be59fd","Type":"ContainerStarted","Data":"7e50dcc5e9bab2384723f21ee509cf18d10e7d7a10663a59b1ae27b7924102ea"} Nov 02 08:14:55.294899 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:55.294859 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" event={"ID":"74e96d56-8e33-4351-814b-3d9781be59fd","Type":"ContainerStarted","Data":"146505a65ce51814a43f2b188849e6dfdb720fd6e43e5bb801048991a481d74d"} Nov 02 08:14:55.295296 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:55.294931 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:14:55.402976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:55.402894 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:55.410770 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:14:55.410719 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" podStartSLOduration=1.546056218 podStartE2EDuration="2.410707476s" totalImagesPullingTime="864.651258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:14:55.410336418 +0000 UTC m=+493.963840634" watchObservedRunningTime="2026-11-02 08:14:55.410707476 +0000 UTC m=+493.964211673" Nov 02 08:15:00.157421 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.157379 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf"] Nov 02 08:15:00.162674 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.162650 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:00.260051 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.260007 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:15:00.260051 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.260042 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:15:00.260306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.260048 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:15:00.261936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.261914 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf"] Nov 02 08:15:00.289553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.289527 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893455-twgbf\" (UID: \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:00.289705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.289560 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmcf8\" (UniqueName: \"kubernetes.io/projected/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-kube-api-access-bmcf8\") pod \"maas-api-key-cleanup-29893455-twgbf\" (UID: \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:00.390615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.390575 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893455-twgbf\" (UID: \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:00.390615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.390618 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmcf8\" (UniqueName: \"kubernetes.io/projected/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-kube-api-access-bmcf8\") pod \"maas-api-key-cleanup-29893455-twgbf\" (UID: \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:00.391298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.391280 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893455-twgbf\" (UID: \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:00.443607 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.443580 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmcf8\" (UniqueName: \"kubernetes.io/projected/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-kube-api-access-bmcf8\") pod \"maas-api-key-cleanup-29893455-twgbf\" (UID: \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:00.452193 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.452149 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:00.474071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.474034 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:00.838088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:00.838065 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf"] Nov 02 08:15:00.840316 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:15:00.840286 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc84f0a3_fc2a_4321_bf3b_15bc3401ed24.slice/crio-782f0b639b9cf61303395be4d96d26b50a3c83f887388b3f08152a2ae72d01fc WatchSource:0}: Error finding container 782f0b639b9cf61303395be4d96d26b50a3c83f887388b3f08152a2ae72d01fc: Status 404 returned error can't find the container with id 782f0b639b9cf61303395be4d96d26b50a3c83f887388b3f08152a2ae72d01fc Nov 02 08:15:01.326850 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:01.326803 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" event={"ID":"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24","Type":"ContainerStarted","Data":"782f0b639b9cf61303395be4d96d26b50a3c83f887388b3f08152a2ae72d01fc"} Nov 02 08:15:05.347879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:05.347839 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" event={"ID":"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24","Type":"ContainerStarted","Data":"dcc0778e554ab17089758f6361cb1cd79ae38390e843c493042a1568f2809e74"} Nov 02 08:15:05.456873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:05.456810 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" podStartSLOduration=1.498840867 podStartE2EDuration="5.456796246s" totalImagesPullingTime="3.957955379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:05.454381116 +0000 UTC m=+504.007885317" watchObservedRunningTime="2026-11-02 08:15:05.456796246 +0000 UTC m=+504.010300443" Nov 02 08:15:06.099942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.099910 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:06.302585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.302554 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-c9959c9bb-l695c" Nov 02 08:15:06.307530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.307502 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:06.354985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.354896 6160 generic.go:356] "Generic (PLEG): container finished" podID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerID="dcc0778e554ab17089758f6361cb1cd79ae38390e843c493042a1568f2809e74" exitCode=60 Nov 02 08:15:06.355375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.354982 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" event={"ID":"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24","Type":"ContainerDied","Data":"dcc0778e554ab17089758f6361cb1cd79ae38390e843c493042a1568f2809e74"} Nov 02 08:15:06.355375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.355143 6160 scope.go:122] "RemoveContainer" containerID="dcc0778e554ab17089758f6361cb1cd79ae38390e843c493042a1568f2809e74" Nov 02 08:15:06.454727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.454696 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2kkhq"] Nov 02 08:15:06.454914 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.454895 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" podUID="391cad15-d20a-4d7c-a118-108d43622ac2" containerName="manager" containerID="cri-o://0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789" gracePeriod=30 Nov 02 08:15:06.732520 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.732494 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:15:06.740680 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.740660 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391cad15-d20a-4d7c-a118-108d43622ac2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/391cad15-d20a-4d7c-a118-108d43622ac2-metrics-certs\") pod \"391cad15-d20a-4d7c-a118-108d43622ac2\" (UID: \"391cad15-d20a-4d7c-a118-108d43622ac2\") " Nov 02 08:15:06.740800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.740721 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391cad15-d20a-4d7c-a118-108d43622ac2-kube-api-access-xkjct\" (UniqueName: \"kubernetes.io/projected/391cad15-d20a-4d7c-a118-108d43622ac2-kube-api-access-xkjct\") pod \"391cad15-d20a-4d7c-a118-108d43622ac2\" (UID: \"391cad15-d20a-4d7c-a118-108d43622ac2\") " Nov 02 08:15:06.742933 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.742903 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391cad15-d20a-4d7c-a118-108d43622ac2-kube-api-access-xkjct" pod "391cad15-d20a-4d7c-a118-108d43622ac2" (UID: "391cad15-d20a-4d7c-a118-108d43622ac2"). InnerVolumeSpecName "kube-api-access-xkjct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:15:06.742933 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.742915 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391cad15-d20a-4d7c-a118-108d43622ac2-metrics-certs" pod "391cad15-d20a-4d7c-a118-108d43622ac2" (UID: "391cad15-d20a-4d7c-a118-108d43622ac2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:15:06.842061 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.842026 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkjct\" (UniqueName: \"kubernetes.io/projected/391cad15-d20a-4d7c-a118-108d43622ac2-kube-api-access-xkjct\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:15:06.842061 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:06.842062 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/391cad15-d20a-4d7c-a118-108d43622ac2-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:15:07.362349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.362262 6160 generic.go:356] "Generic (PLEG): container finished" podID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerID="2010891c8dc033b5128f4b8142f51111e40b17a688cea9ff5d1bca17947e0930" exitCode=60 Nov 02 08:15:07.362746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.362348 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" event={"ID":"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24","Type":"ContainerDied","Data":"2010891c8dc033b5128f4b8142f51111e40b17a688cea9ff5d1bca17947e0930"} Nov 02 08:15:07.362746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.362405 6160 scope.go:122] "RemoveContainer" containerID="dcc0778e554ab17089758f6361cb1cd79ae38390e843c493042a1568f2809e74" Nov 02 08:15:07.362746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.362630 6160 scope.go:122] "RemoveContainer" containerID="2010891c8dc033b5128f4b8142f51111e40b17a688cea9ff5d1bca17947e0930" Nov 02 08:15:07.362876 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:15:07.362849 6160 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893455-twgbf_odh-ai-gateway-infra(fc84f0a3-fc2a-4321-bf3b-15bc3401ed24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" Nov 02 08:15:07.363671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.363652 6160 generic.go:356] "Generic (PLEG): container finished" podID="391cad15-d20a-4d7c-a118-108d43622ac2" containerID="0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789" exitCode=0 Nov 02 08:15:07.363763 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.363708 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" Nov 02 08:15:07.363763 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.363738 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" event={"ID":"391cad15-d20a-4d7c-a118-108d43622ac2","Type":"ContainerDied","Data":"0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789"} Nov 02 08:15:07.363872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.363778 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2kkhq" event={"ID":"391cad15-d20a-4d7c-a118-108d43622ac2","Type":"ContainerDied","Data":"77ed19c8bfe393c2f5145c7fb80159a94953de1cc288b5948f550eeb283f5b9e"} Nov 02 08:15:07.378866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.378842 6160 scope.go:122] "RemoveContainer" containerID="0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789" Nov 02 08:15:07.386836 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.386821 6160 scope.go:122] "RemoveContainer" containerID="0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789" Nov 02 08:15:07.387091 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:15:07.387073 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789\": container with ID starting with 0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789 not found: ID does not exist" containerID="0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789" Nov 02 08:15:07.387141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.387098 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789"} err="failed to get container status \"0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789\": rpc error: code = NotFound desc = could not find container \"0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789\": container with ID starting with 0bc4bd123958fa2ef5c02211ebcbc054ffdec0a3aa213ecd25ca97d903639789 not found: ID does not exist" Nov 02 08:15:07.571322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.571292 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2kkhq"] Nov 02 08:15:07.575700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.575675 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2kkhq"] Nov 02 08:15:07.582012 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:07.581990 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391cad15-d20a-4d7c-a118-108d43622ac2" path="/var/lib/kubelet/pods/391cad15-d20a-4d7c-a118-108d43622ac2/volumes" Nov 02 08:15:08.096866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:08.096822 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:08.371550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:08.371476 6160 scope.go:122] "RemoveContainer" containerID="2010891c8dc033b5128f4b8142f51111e40b17a688cea9ff5d1bca17947e0930" Nov 02 08:15:08.371901 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:15:08.371670 6160 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893455-twgbf_odh-ai-gateway-infra(fc84f0a3-fc2a-4321-bf3b-15bc3401ed24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" Nov 02 08:15:08.489845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:08.489809 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:10.520309 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:15:10.520269 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:15:19.624589 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:19.624544 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:19.791445 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:19.791414 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:21.083046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:21.083013 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:21.580854 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:21.580817 6160 scope.go:122] "RemoveContainer" containerID="2010891c8dc033b5128f4b8142f51111e40b17a688cea9ff5d1bca17947e0930" Nov 02 08:15:22.441105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:22.441075 6160 generic.go:356] "Generic (PLEG): container finished" podID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerID="abe32fae8708ede2bab64e7584acafd5c69f82b5478985304ca41f99c31b7e65" exitCode=60 Nov 02 08:15:22.441595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:22.441153 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" event={"ID":"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24","Type":"ContainerDied","Data":"abe32fae8708ede2bab64e7584acafd5c69f82b5478985304ca41f99c31b7e65"} Nov 02 08:15:22.441595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:22.441243 6160 scope.go:122] "RemoveContainer" containerID="2010891c8dc033b5128f4b8142f51111e40b17a688cea9ff5d1bca17947e0930" Nov 02 08:15:22.441724 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:22.441598 6160 scope.go:122] "RemoveContainer" containerID="abe32fae8708ede2bab64e7584acafd5c69f82b5478985304ca41f99c31b7e65" Nov 02 08:15:22.441825 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:15:22.441802 6160 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893455-twgbf_odh-ai-gateway-infra(fc84f0a3-fc2a-4321-bf3b-15bc3401ed24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" Nov 02 08:15:23.559015 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:23.558956 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf"] Nov 02 08:15:23.680343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:23.680317 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:23.784071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:23.784040 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-maas-api-service-ca\") pod \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\" (UID: \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\") " Nov 02 08:15:23.784270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:23.784099 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-kube-api-access-bmcf8\" (UniqueName: \"kubernetes.io/projected/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-kube-api-access-bmcf8\") pod \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\" (UID: \"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24\") " Nov 02 08:15:23.784449 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:23.784424 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-maas-api-service-ca" pod "fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" (UID: "fc84f0a3-fc2a-4321-bf3b-15bc3401ed24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:15:23.786988 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:23.786958 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-kube-api-access-bmcf8" pod "fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" (UID: "fc84f0a3-fc2a-4321-bf3b-15bc3401ed24"). InnerVolumeSpecName "kube-api-access-bmcf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:15:23.885624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:23.885515 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-maas-api-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:15:23.885624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:23.885564 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmcf8\" (UniqueName: \"kubernetes.io/projected/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24-kube-api-access-bmcf8\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:15:24.454141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:24.454106 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" Nov 02 08:15:24.454141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:24.454121 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf" event={"ID":"fc84f0a3-fc2a-4321-bf3b-15bc3401ed24","Type":"ContainerDied","Data":"782f0b639b9cf61303395be4d96d26b50a3c83f887388b3f08152a2ae72d01fc"} Nov 02 08:15:24.454399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:24.454199 6160 scope.go:122] "RemoveContainer" containerID="abe32fae8708ede2bab64e7584acafd5c69f82b5478985304ca41f99c31b7e65" Nov 02 08:15:24.628133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:24.628097 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf"] Nov 02 08:15:24.632396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:24.632370 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-twgbf"] Nov 02 08:15:25.581519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:25.581486 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" path="/var/lib/kubelet/pods/fc84f0a3-fc2a-4321-bf3b-15bc3401ed24/volumes" Nov 02 08:15:35.881150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881117 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc"] Nov 02 08:15:35.888492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881434 6160 cpu_manager.go:414] "containerMap: removing container" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerName="cleanup" Nov 02 08:15:35.888492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881444 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerName="cleanup" Nov 02 08:15:35.888492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881467 6160 cpu_manager.go:414] "containerMap: removing container" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerName="cleanup" Nov 02 08:15:35.888492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881472 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerName="cleanup" Nov 02 08:15:35.888492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881493 6160 cpu_manager.go:414] "containerMap: removing container" podUID="391cad15-d20a-4d7c-a118-108d43622ac2" containerName="manager" Nov 02 08:15:35.888492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881497 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391cad15-d20a-4d7c-a118-108d43622ac2" containerName="manager" Nov 02 08:15:35.888492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881547 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerName="cleanup" Nov 02 08:15:35.888492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881558 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="391cad15-d20a-4d7c-a118-108d43622ac2" containerName="manager" Nov 02 08:15:35.888492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.881565 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerName="cleanup" Nov 02 08:15:35.889047 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.889031 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:35.942430 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.942396 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc"] Nov 02 08:15:35.984257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.984217 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 08:15:35.984448 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.984294 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 08:15:35.984448 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.984307 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 08:15:35.984448 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.984333 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q6gfs\"" Nov 02 08:15:35.987681 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.987657 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:35.987783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.987723 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:35.987783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.987766 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:35.987908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.987884 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:35.987975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.987944 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c213051c-75c6-4e21-95cb-3870a450b435-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:35.988033 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.988004 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rgqr\" (UniqueName: \"kubernetes.io/projected/c213051c-75c6-4e21-95cb-3870a450b435-kube-api-access-8rgqr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:35.988918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:35.988895 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:36.088602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.088564 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.088602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.088602 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.088841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.088790 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.088910 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.088841 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c213051c-75c6-4e21-95cb-3870a450b435-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.088910 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.088861 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rgqr\" (UniqueName: \"kubernetes.io/projected/c213051c-75c6-4e21-95cb-3870a450b435-kube-api-access-8rgqr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.089019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.088970 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.089079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.089042 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.089159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.089136 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.089346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.089327 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.091055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.091038 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c213051c-75c6-4e21-95cb-3870a450b435-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.091480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.091461 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c213051c-75c6-4e21-95cb-3870a450b435-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.141918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.141849 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rgqr\" (UniqueName: \"kubernetes.io/projected/c213051c-75c6-4e21-95cb-3870a450b435-kube-api-access-8rgqr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc\" (UID: \"c213051c-75c6-4e21-95cb-3870a450b435\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.201854 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.201813 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:36.565765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:36.565741 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc"] Nov 02 08:15:36.568428 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:15:36.568406 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc213051c_75c6_4e21_95cb_3870a450b435.slice/crio-4032291be327a7539409249d960b697b86fa416ed5895bb230fb601d8255590b WatchSource:0}: Error finding container 4032291be327a7539409249d960b697b86fa416ed5895bb230fb601d8255590b: Status 404 returned error can't find the container with id 4032291be327a7539409249d960b697b86fa416ed5895bb230fb601d8255590b Nov 02 08:15:37.523859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:37.523782 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" event={"ID":"c213051c-75c6-4e21-95cb-3870a450b435","Type":"ContainerStarted","Data":"4032291be327a7539409249d960b697b86fa416ed5895bb230fb601d8255590b"} Nov 02 08:15:39.535185 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:39.535128 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" event={"ID":"c213051c-75c6-4e21-95cb-3870a450b435","Type":"ContainerStarted","Data":"687496c98d0777809613e952355c9fd78a5ec84714d1e1f2516b703764667f9e"} Nov 02 08:15:39.642076 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:39.642027 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" podStartSLOduration=2.631954624 podStartE2EDuration="4.642012905s" totalImagesPullingTime="2.010058281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:39.640338792 +0000 UTC m=+538.193843008" watchObservedRunningTime="2026-11-02 08:15:39.642012905 +0000 UTC m=+538.195517106" Nov 02 08:15:40.202083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:40.202048 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:40.520294 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:15:40.520202 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:15:41.502667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:41.502631 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:45.286620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:45.286582 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:46.202558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:46.202526 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:46.215496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:46.215468 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:46.581498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:46.581410 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc" Nov 02 08:15:49.295804 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.295766 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l"] Nov 02 08:15:49.296214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.296114 6160 cpu_manager.go:414] "containerMap: removing container" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerName="cleanup" Nov 02 08:15:49.296214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.296124 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerName="cleanup" Nov 02 08:15:49.296214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.296209 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc84f0a3-fc2a-4321-bf3b-15bc3401ed24" containerName="cleanup" Nov 02 08:15:49.300314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.300296 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.354252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.354214 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l"] Nov 02 08:15:49.398018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.397988 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 08:15:49.399248 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.399226 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.399495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.399298 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vfrg\" (UniqueName: \"kubernetes.io/projected/ec12dd43-98b4-47b7-adb0-c6aa56108563-kube-api-access-7vfrg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.399495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.399352 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.399495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.399435 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.399495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.399464 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec12dd43-98b4-47b7-adb0-c6aa56108563-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.399654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.399549 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.405291 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.405263 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:49.500549 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.500515 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.500549 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.500550 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec12dd43-98b4-47b7-adb0-c6aa56108563-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.500802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.500579 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.500802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.500702 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.500955 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.500798 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vfrg\" (UniqueName: \"kubernetes.io/projected/ec12dd43-98b4-47b7-adb0-c6aa56108563-kube-api-access-7vfrg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.500955 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.500839 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.500955 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.500924 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.501113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.500981 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.501202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.501167 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.503094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.503077 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec12dd43-98b4-47b7-adb0-c6aa56108563-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.503366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.503347 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec12dd43-98b4-47b7-adb0-c6aa56108563-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.554336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.554269 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vfrg\" (UniqueName: \"kubernetes.io/projected/ec12dd43-98b4-47b7-adb0-c6aa56108563-kube-api-access-7vfrg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rkx8l\" (UID: \"ec12dd43-98b4-47b7-adb0-c6aa56108563\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.612458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.612421 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:49.977431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:49.977401 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l"] Nov 02 08:15:49.979777 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:15:49.979752 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec12dd43_98b4_47b7_adb0_c6aa56108563.slice/crio-ffa371689893d1bd68a6f066622d6f9c5fb50cb97280cb8a409a518b71f94d7e WatchSource:0}: Error finding container ffa371689893d1bd68a6f066622d6f9c5fb50cb97280cb8a409a518b71f94d7e: Status 404 returned error can't find the container with id ffa371689893d1bd68a6f066622d6f9c5fb50cb97280cb8a409a518b71f94d7e Nov 02 08:15:50.589489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:50.589455 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" event={"ID":"ec12dd43-98b4-47b7-adb0-c6aa56108563","Type":"ContainerStarted","Data":"ffa371689893d1bd68a6f066622d6f9c5fb50cb97280cb8a409a518b71f94d7e"} Nov 02 08:15:51.595773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:51.595738 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" event={"ID":"ec12dd43-98b4-47b7-adb0-c6aa56108563","Type":"ContainerStarted","Data":"f70a90d11cda9caa2aa5edff19efc93d79dcc28fa304451f7725d2e3a632a05e"} Nov 02 08:15:51.613053 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:51.613031 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:51.704996 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:51.704942 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" podStartSLOduration=2.122088474 podStartE2EDuration="2.704926922s" totalImagesPullingTime="582.838448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:51.702885396 +0000 UTC m=+550.256389592" watchObservedRunningTime="2026-11-02 08:15:51.704926922 +0000 UTC m=+550.258431120" Nov 02 08:15:52.956757 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:52.956723 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h"] Nov 02 08:15:52.960099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:52.960081 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.055010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.054980 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 08:15:53.057199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.057149 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h"] Nov 02 08:15:53.080139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.080104 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:53.131829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.131801 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.132010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.131839 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.132010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.131865 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8qt\" (UniqueName: \"kubernetes.io/projected/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-kube-api-access-kb8qt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.132010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.131987 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.132115 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.132021 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.132150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.132119 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.233352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.233261 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.233352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.233316 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.233352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.233341 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.233600 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.233365 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8qt\" (UniqueName: \"kubernetes.io/projected/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-kube-api-access-kb8qt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.233600 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.233395 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.233600 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.233418 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.233795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.233772 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.233849 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.233780 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.233849 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.233832 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.235859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.235834 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.236103 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.236086 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.287237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.287207 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8qt\" (UniqueName: \"kubernetes.io/projected/3d35ef8c-274f-4e8d-97f4-c0960dd718b6-kube-api-access-kb8qt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h\" (UID: \"3d35ef8c-274f-4e8d-97f4-c0960dd718b6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.572687 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.572599 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:53.937989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:53.937965 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h"] Nov 02 08:15:53.940299 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:15:53.940274 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d35ef8c_274f_4e8d_97f4_c0960dd718b6.slice/crio-3afd1670e1802be218f51f7df9486e4a9b57451510801e97b5288cb316682ab8 WatchSource:0}: Error finding container 3afd1670e1802be218f51f7df9486e4a9b57451510801e97b5288cb316682ab8: Status 404 returned error can't find the container with id 3afd1670e1802be218f51f7df9486e4a9b57451510801e97b5288cb316682ab8 Nov 02 08:15:54.613065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:54.613037 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" event={"ID":"3d35ef8c-274f-4e8d-97f4-c0960dd718b6","Type":"ContainerStarted","Data":"3afd1670e1802be218f51f7df9486e4a9b57451510801e97b5288cb316682ab8"} Nov 02 08:15:55.619698 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:55.619658 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" event={"ID":"3d35ef8c-274f-4e8d-97f4-c0960dd718b6","Type":"ContainerStarted","Data":"b9fd0fd528c52ba4cc7054ccf21c9926175b642dbdb43cdbe42a250999756d40"} Nov 02 08:15:55.717063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:55.717024 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:55.730306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:55.730252 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" podStartSLOduration=3.108116067 podStartE2EDuration="3.730235626s" totalImagesPullingTime="622.119559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:55.728794566 +0000 UTC m=+554.282298762" watchObservedRunningTime="2026-11-02 08:15:55.730235626 +0000 UTC m=+554.283739829" Nov 02 08:15:55.958071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:55.958029 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:56.572842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:56.572798 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:15:59.613426 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:59.613397 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:59.626110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:59.626085 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:15:59.654568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:15:59.654542 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rkx8l" Nov 02 08:16:00.242619 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.242588 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww"] Nov 02 08:16:00.246316 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.246298 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.295442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.295410 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 08:16:00.345709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.345666 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww"] Nov 02 08:16:00.372707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.372671 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:00.395746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.395717 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.395870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.395783 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.395870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.395857 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wpt\" (UniqueName: \"kubernetes.io/projected/50909652-5fa6-4f37-9d22-6122af29d8b1-kube-api-access-w7wpt\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.395951 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.395922 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50909652-5fa6-4f37-9d22-6122af29d8b1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.395993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.395952 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.395993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.395971 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.497150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.497053 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.497150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.497121 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.497389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.497201 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.497389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.497229 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wpt\" (UniqueName: \"kubernetes.io/projected/50909652-5fa6-4f37-9d22-6122af29d8b1-kube-api-access-w7wpt\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.497389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.497297 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50909652-5fa6-4f37-9d22-6122af29d8b1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.497389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.497324 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.497667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.497641 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.497738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.497648 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.497738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.497683 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.499513 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.499491 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50909652-5fa6-4f37-9d22-6122af29d8b1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.499855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.499834 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50909652-5fa6-4f37-9d22-6122af29d8b1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.563695 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.563657 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wpt\" (UniqueName: \"kubernetes.io/projected/50909652-5fa6-4f37-9d22-6122af29d8b1-kube-api-access-w7wpt\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww\" (UID: \"50909652-5fa6-4f37-9d22-6122af29d8b1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:00.831712 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.831617 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:00.859476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:00.859453 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:01.230105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:01.230071 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww"] Nov 02 08:16:01.233639 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:16:01.233612 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50909652_5fa6_4f37_9d22_6122af29d8b1.slice/crio-a29c05c25bae2d1d1bb5157c2bc56f0850583278387058bc519fc627000b9a77 WatchSource:0}: Error finding container a29c05c25bae2d1d1bb5157c2bc56f0850583278387058bc519fc627000b9a77: Status 404 returned error can't find the container with id a29c05c25bae2d1d1bb5157c2bc56f0850583278387058bc519fc627000b9a77 Nov 02 08:16:01.653753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:01.653667 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" event={"ID":"50909652-5fa6-4f37-9d22-6122af29d8b1","Type":"ContainerStarted","Data":"a29c05c25bae2d1d1bb5157c2bc56f0850583278387058bc519fc627000b9a77"} Nov 02 08:16:02.660500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:02.660460 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" event={"ID":"50909652-5fa6-4f37-9d22-6122af29d8b1","Type":"ContainerStarted","Data":"f606962c3812abf760156bff9be0fab5d07c2a7bb780a68491b18742a4d1ff7e"} Nov 02 08:16:02.769057 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:02.769001 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" podStartSLOduration=2.238608954 podStartE2EDuration="2.768986396s" totalImagesPullingTime="530.377442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:02.767632798 +0000 UTC m=+561.321136995" watchObservedRunningTime="2026-11-02 08:16:02.768986396 +0000 UTC m=+561.322490599" Nov 02 08:16:02.859836 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:02.859805 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:03.573375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:03.573332 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:16:03.590280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:03.590254 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:16:03.676639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:03.676609 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h" Nov 02 08:16:03.727790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:03.727752 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:05.461653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.461619 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd"] Nov 02 08:16:05.465221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.465203 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.513717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.513685 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 08:16:05.542900 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.542866 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.543082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.542911 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.543082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.542938 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.543082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.543060 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.543213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.543096 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptf4\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-kube-api-access-5ptf4\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.543213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.543122 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.543286 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.543229 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.543286 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.543253 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.543286 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.543277 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.560097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.560069 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zmgsp\"" Nov 02 08:16:05.562461 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.562439 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd"] Nov 02 08:16:05.643967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.643931 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.643967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.643964 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.643987 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptf4\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-kube-api-access-5ptf4\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.644021 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.644053 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.644098 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644499 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.644411 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644560 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.644519 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644560 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.644525 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.644589 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644791 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.644768 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.644906 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.644863 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.645232 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.645208 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.645404 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.645381 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.646659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.646638 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.646948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.646930 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.697405 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.697372 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.697651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.697631 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptf4\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-kube-api-access-5ptf4\") pod \"maas-default-gateway-openshift-default-769df7846c-skrdd\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:05.777712 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:05.777621 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:06.194733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:06.194704 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd"] Nov 02 08:16:06.196464 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:16:06.196435 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c2349b_4ccf_4e8b_8507_1b630e65fa30.slice/crio-d812d59808eaef8bef8bdb5cecd119ca994e59f9a5131e9a934f3fec12d20ea6 WatchSource:0}: Error finding container d812d59808eaef8bef8bdb5cecd119ca994e59f9a5131e9a934f3fec12d20ea6: Status 404 returned error can't find the container with id d812d59808eaef8bef8bdb5cecd119ca994e59f9a5131e9a934f3fec12d20ea6 Nov 02 08:16:06.683191 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:06.683149 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" event={"ID":"23c2349b-4ccf-4e8b-8507-1b630e65fa30","Type":"ContainerStarted","Data":"d812d59808eaef8bef8bdb5cecd119ca994e59f9a5131e9a934f3fec12d20ea6"} Nov 02 08:16:10.520050 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:16:10.520006 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:16:10.529989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:10.529955 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:16:10.530082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:10.530040 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:16:10.530082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:10.530074 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:16:10.705945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:10.705902 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" event={"ID":"23c2349b-4ccf-4e8b-8507-1b630e65fa30","Type":"ContainerStarted","Data":"a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450"} Nov 02 08:16:10.778330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:10.778243 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:10.860119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:10.860088 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:10.860945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:10.860900 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" podStartSLOduration=1.530200812 podStartE2EDuration="5.860886401s" totalImagesPullingTime="4.330685589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:10.81403174 +0000 UTC m=+569.367535937" watchObservedRunningTime="2026-11-02 08:16:10.860886401 +0000 UTC m=+569.414390597" Nov 02 08:16:10.872992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:10.872967 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:11.745050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:11.745022 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww" Nov 02 08:16:11.780343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:11.780303 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-skrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 08:16:11.780502 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:11.780356 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 08:16:12.778929 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:12.778896 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-skrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 08:16:12.779350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:12.778957 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 08:16:13.778275 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:13.778242 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-skrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 08:16:13.778463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:13.778295 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 08:16:14.778284 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:14.778245 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-skrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 08:16:14.778774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:14.778318 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 08:16:15.778078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:15.778042 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-skrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 08:16:15.778320 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:15.778105 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 08:16:16.778249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:16.778207 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-skrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 08:16:16.778432 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:16.778278 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 08:16:17.778859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:17.778826 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-skrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 08:16:17.779278 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:17.778879 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 08:16:18.823887 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:18.823849 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:18.824388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:18.824278 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:18.825037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:18.825007 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:16:32.213807 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:32.213763 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:33.442169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:33.442126 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:33.655159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:33.655123 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:34.944944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:34.944906 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:35.225856 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.225768 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:35.737090 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.737059 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn"] Nov 02 08:16:35.740790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.740770 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.837644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.837611 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn"] Nov 02 08:16:35.851040 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.851005 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.851207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.851107 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ml2x\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-kube-api-access-2ml2x\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.851207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.851139 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.851207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.851185 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.851353 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.851208 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.851397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.851368 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.851397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.851390 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.851500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.851486 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.851549 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.851508 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952059 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml2x\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-kube-api-access-2ml2x\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952115 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952141 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952161 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952235 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952252 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952278 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952295 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952331 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952641 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952683 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.953189 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952737 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.953189 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.952814 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.953189 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.953067 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.954723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.954703 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:35.954949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:35.954931 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:36.006406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.006328 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:36.006406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.006377 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ml2x\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-kube-api-access-2ml2x\") pod \"maas-default-gateway-openshift-default-769df7846c-8t2kn\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:36.053417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.053375 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:36.482723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.480162 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn"] Nov 02 08:16:36.512464 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:16:36.512433 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d462b83_c9b7_4cba_8aed_16fba5fffa52.slice/crio-40f534665b758b424bf1a1ee45c8088cf55b6147c1fb01ce98df704a9447c5cf WatchSource:0}: Error finding container 40f534665b758b424bf1a1ee45c8088cf55b6147c1fb01ce98df704a9447c5cf: Status 404 returned error can't find the container with id 40f534665b758b424bf1a1ee45c8088cf55b6147c1fb01ce98df704a9447c5cf Nov 02 08:16:36.514512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.514475 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:16:36.514618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.514561 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:16:36.514676 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.514631 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:16:36.862950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.862857 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" event={"ID":"6d462b83-c9b7-4cba-8aed-16fba5fffa52","Type":"ContainerStarted","Data":"9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a"} Nov 02 08:16:36.862950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.862917 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" event={"ID":"6d462b83-c9b7-4cba-8aed-16fba5fffa52","Type":"ContainerStarted","Data":"40f534665b758b424bf1a1ee45c8088cf55b6147c1fb01ce98df704a9447c5cf"} Nov 02 08:16:36.979601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:36.979550 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podStartSLOduration=1.979537178 podStartE2EDuration="1.979537178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:36.978258283 +0000 UTC m=+595.531762480" watchObservedRunningTime="2026-11-02 08:16:36.979537178 +0000 UTC m=+595.533041443" Nov 02 08:16:37.054138 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:37.054099 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:37.055620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:37.055594 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8t2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 08:16:37.055744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:37.055667 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 08:16:38.053736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:38.053701 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8t2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 08:16:38.054246 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:38.053759 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 08:16:39.054726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:39.054686 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8t2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 08:16:39.055123 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:39.054742 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 08:16:40.054752 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:40.054715 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8t2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 08:16:40.055149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:40.054788 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 08:16:40.541467 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:16:40.541433 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:16:41.054498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:41.054462 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8t2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 08:16:41.054670 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:41.054531 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 08:16:41.645466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:41.645434 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:16:41.646762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:41.646739 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:16:42.054044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:42.054009 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8t2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 08:16:42.054281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:42.054064 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 08:16:43.054045 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:43.054001 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8t2kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 08:16:43.054559 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:43.054073 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 08:16:44.727388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:44.727351 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:44.727893 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:44.727750 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:16:44.728700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:16:44.728680 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:17:01.179677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:01.179637 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:02.078893 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:02.078846 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:02.303444 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:02.303408 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:02.504015 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:02.503980 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:02.696687 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:02.696649 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:02.890764 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:02.890678 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:03.057958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.057913 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:03.251275 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.251231 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:03.547386 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.547292 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:03.673588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.673555 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm"] Nov 02 08:17:03.676592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.676577 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.725909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.725880 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-53cc8e62-gw-openshift-default-dockercfg-pjkkr\"" Nov 02 08:17:03.737316 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.737280 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-envoy\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.737481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.737329 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3242831-c651-4f6b-953f-32726b062cc4-istio-podinfo\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.737481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.737365 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-credential-socket\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.737481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.737398 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-socket\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.737481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.737473 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3242831-c651-4f6b-953f-32726b062cc4-istiod-ca-cert\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.737705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.737562 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4xd\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-kube-api-access-rb4xd\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.737705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.737618 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-istio-token\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.737786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.737710 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-certs\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.737830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.737788 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-data\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.773764 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.773730 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm"] Nov 02 08:17:03.804108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.804025 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:03.839157 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839128 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-istio-token\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839162 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-certs\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839221 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-data\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839294 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-envoy\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839445 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839315 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3242831-c651-4f6b-953f-32726b062cc4-istio-podinfo\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839445 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839337 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-credential-socket\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839445 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839421 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-socket\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839593 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839470 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3242831-c651-4f6b-953f-32726b062cc4-istiod-ca-cert\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839593 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839507 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4xd\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-kube-api-access-rb4xd\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839689 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839666 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-certs\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839717 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-data\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.839799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.839768 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-credential-socket\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.840151 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.840131 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3242831-c651-4f6b-953f-32726b062cc4-istiod-ca-cert\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.840402 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.840381 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-socket\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.841695 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.841678 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-envoy\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.841817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.841800 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3242831-c651-4f6b-953f-32726b062cc4-istio-podinfo\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.903590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.903560 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4xd\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-kube-api-access-rb4xd\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.905866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.905831 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-istio-token\") pod \"e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:03.989663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:03.989629 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:04.046749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:04.046720 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:04.414069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:04.414037 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm"] Nov 02 08:17:04.416454 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:04.416426 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3242831_c651_4f6b_953f_32726b062cc4.slice/crio-b00f61ad322ab486fe9046af82322e17fb048d96f4db64a935e54702a926fb25 WatchSource:0}: Error finding container b00f61ad322ab486fe9046af82322e17fb048d96f4db64a935e54702a926fb25: Status 404 returned error can't find the container with id b00f61ad322ab486fe9046af82322e17fb048d96f4db64a935e54702a926fb25 Nov 02 08:17:04.418544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:04.418508 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:17:04.418650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:04.418590 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:17:04.418650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:04.418631 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:17:05.004730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:05.004693 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" event={"ID":"c3242831-c651-4f6b-953f-32726b062cc4","Type":"ContainerStarted","Data":"ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda"} Nov 02 08:17:05.005125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:05.004748 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" event={"ID":"c3242831-c651-4f6b-953f-32726b062cc4","Type":"ContainerStarted","Data":"b00f61ad322ab486fe9046af82322e17fb048d96f4db64a935e54702a926fb25"} Nov 02 08:17:05.118010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:05.117940 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" podStartSLOduration=2.11792114 podStartE2EDuration="2.11792114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:05.114450546 +0000 UTC m=+623.667954766" watchObservedRunningTime="2026-11-02 08:17:05.11792114 +0000 UTC m=+623.671425339" Nov 02 08:17:05.772425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:05.772385 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:05.990226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:05.990159 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:05.995297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:05.995273 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:06.010324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:06.010295 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:06.011360 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:06.011343 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:06.020856 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:06.020826 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:06.207055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:06.207019 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:06.799319 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:06.799283 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:07.011772 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:07.011725 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:08.100663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:08.100627 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:09.075665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.075632 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:09.303726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.303692 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:09.517416 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.517381 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz"] Nov 02 08:17:09.521127 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.521110 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:09.527558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.527532 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:09.573426 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.573401 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2b62c067\"" Nov 02 08:17:09.595780 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.595745 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963df6f-c03e-407e-8d8d-355141c73c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz\" (UID: \"c963df6f-c03e-407e-8d8d-355141c73c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:09.595970 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.595941 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stv98\" (UniqueName: \"kubernetes.io/projected/c963df6f-c03e-407e-8d8d-355141c73c40-kube-api-access-stv98\") pod \"payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz\" (UID: \"c963df6f-c03e-407e-8d8d-355141c73c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:09.621214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.618531 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2b62c067-dockercfg-xgkjr\"" Nov 02 08:17:09.621214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.620741 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz"] Nov 02 08:17:09.622728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.622699 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c"] Nov 02 08:17:09.626426 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.626405 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:09.692300 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.692268 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c"] Nov 02 08:17:09.701647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.701611 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c\" (UID: \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\") " pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:09.701824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.701694 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963df6f-c03e-407e-8d8d-355141c73c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz\" (UID: \"c963df6f-c03e-407e-8d8d-355141c73c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:09.701937 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.701909 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwr2\" (UniqueName: \"kubernetes.io/projected/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-kube-api-access-mkwr2\") pod \"payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c\" (UID: \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\") " pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:09.702164 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.702068 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stv98\" (UniqueName: \"kubernetes.io/projected/c963df6f-c03e-407e-8d8d-355141c73c40-kube-api-access-stv98\") pod \"payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz\" (UID: \"c963df6f-c03e-407e-8d8d-355141c73c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:09.702595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.702575 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963df6f-c03e-407e-8d8d-355141c73c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz\" (UID: \"c963df6f-c03e-407e-8d8d-355141c73c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:09.724075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.724041 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:09.755619 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.755593 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stv98\" (UniqueName: \"kubernetes.io/projected/c963df6f-c03e-407e-8d8d-355141c73c40-kube-api-access-stv98\") pod \"payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz\" (UID: \"c963df6f-c03e-407e-8d8d-355141c73c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:09.803655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.803568 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwr2\" (UniqueName: \"kubernetes.io/projected/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-kube-api-access-mkwr2\") pod \"payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c\" (UID: \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\") " pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:09.803655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.803640 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c\" (UID: \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\") " pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:09.804358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.804338 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c\" (UID: \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\") " pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:09.835377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.835343 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:09.864476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.864448 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwr2\" (UniqueName: \"kubernetes.io/projected/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-kube-api-access-mkwr2\") pod \"payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c\" (UID: \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\") " pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:09.940659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:09.940624 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:10.020079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:10.020048 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:10.207590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:10.207562 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz"] Nov 02 08:17:10.210657 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:10.210628 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc963df6f_c03e_407e_8d8d_355141c73c40.slice/crio-23692eb2d97a9cfb4a76548372b464a30ac1db9aada9b95277a2c3deb868be5f WatchSource:0}: Error finding container 23692eb2d97a9cfb4a76548372b464a30ac1db9aada9b95277a2c3deb868be5f: Status 404 returned error can't find the container with id 23692eb2d97a9cfb4a76548372b464a30ac1db9aada9b95277a2c3deb868be5f Nov 02 08:17:10.305487 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:10.305461 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c"] Nov 02 08:17:10.307553 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:10.307526 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae58a388_efd5_4c7d_a0ea_ab2cd7b1d611.slice/crio-1d0be286c137cb3bf7bddca0622d709333e7c4d30389c5562e21d3cf31e93ae5 WatchSource:0}: Error finding container 1d0be286c137cb3bf7bddca0622d709333e7c4d30389c5562e21d3cf31e93ae5: Status 404 returned error can't find the container with id 1d0be286c137cb3bf7bddca0622d709333e7c4d30389c5562e21d3cf31e93ae5 Nov 02 08:17:10.487272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:10.487159 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:10.524514 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:17:10.524474 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:17:10.660141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:10.660111 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:10.826788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:10.826696 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:10.998459 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:10.998415 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:11.040530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:11.040492 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" event={"ID":"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611","Type":"ContainerStarted","Data":"14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c"} Nov 02 08:17:11.040729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:11.040551 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" event={"ID":"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611","Type":"ContainerStarted","Data":"1d0be286c137cb3bf7bddca0622d709333e7c4d30389c5562e21d3cf31e93ae5"} Nov 02 08:17:11.040729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:11.040636 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:11.042072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:11.042042 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" event={"ID":"c963df6f-c03e-407e-8d8d-355141c73c40","Type":"ContainerStarted","Data":"f5056837d28a2b892b10c0dedfe71fb6a37bd72fd607f3356d77ea7cf2352961"} Nov 02 08:17:11.042207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:11.042089 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:11.042207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:11.042105 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" event={"ID":"c963df6f-c03e-407e-8d8d-355141c73c40","Type":"ContainerStarted","Data":"23692eb2d97a9cfb4a76548372b464a30ac1db9aada9b95277a2c3deb868be5f"} Nov 02 08:17:11.148016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:11.147879 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" podStartSLOduration=2.147860566 podStartE2EDuration="2.147860566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:11.14668294 +0000 UTC m=+629.700187140" watchObservedRunningTime="2026-11-02 08:17:11.147860566 +0000 UTC m=+629.701364819" Nov 02 08:17:11.254913 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:11.254852 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" podStartSLOduration=2.254834957 podStartE2EDuration="2.254834957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:11.253332934 +0000 UTC m=+629.806837155" watchObservedRunningTime="2026-11-02 08:17:11.254834957 +0000 UTC m=+629.808339155" Nov 02 08:17:12.215984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.215950 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:12.307756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.307721 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs"] Nov 02 08:17:12.311316 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.311299 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.407614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.407587 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-09e3828d-serving-cert\"" Nov 02 08:17:12.407792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.407662 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:17:12.407792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.407673 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:17:12.407792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.407690 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 08:17:12.407792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.407737 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n86zq\"" Nov 02 08:17:12.409879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.409858 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs"] Nov 02 08:17:12.429864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.429833 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhd9z\" (UniqueName: \"kubernetes.io/projected/2f96b945-6b30-4cf3-82d4-4a8118488cd7-kube-api-access-vhd9z\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.430013 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.429881 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-metrics-certs\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.430013 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.429917 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:12.430013 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.429963 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.430123 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.430054 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f96b945-6b30-4cf3-82d4-4a8118488cd7-openshift-service-ca\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.531619 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.531535 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.531619 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.531582 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f96b945-6b30-4cf3-82d4-4a8118488cd7-openshift-service-ca\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.531813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.531645 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhd9z\" (UniqueName: \"kubernetes.io/projected/2f96b945-6b30-4cf3-82d4-4a8118488cd7-kube-api-access-vhd9z\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.531813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.531671 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-metrics-certs\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.531813 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:17:12.531703 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-serving-cert: secret "maas-api-e2e-ait-09e3828d-serving-cert" not found Nov 02 08:17:12.531813 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:17:12.531785 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls podName:2f96b945-6b30-4cf3-82d4-4a8118488cd7 nodeName:}" failed. No retries permitted until 2026-11-02 08:17:13.031764547 +0000 UTC m=+631.585268749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls") pod "maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" (UID: "2f96b945-6b30-4cf3-82d4-4a8118488cd7") : secret "maas-api-e2e-ait-09e3828d-serving-cert" not found Nov 02 08:17:12.532331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.532309 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f96b945-6b30-4cf3-82d4-4a8118488cd7-openshift-service-ca\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.534301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.534285 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-metrics-certs\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.588068 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.588038 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhd9z\" (UniqueName: \"kubernetes.io/projected/2f96b945-6b30-4cf3-82d4-4a8118488cd7-kube-api-access-vhd9z\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:12.600722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.600694 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:12.802774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.802690 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:12.970159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.970110 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:12.998635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:12.998603 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2"] Nov 02 08:17:13.002278 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.002261 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.037912 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.037876 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:13.040508 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.040484 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls\") pod \"maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:13.060561 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.060482 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2"] Nov 02 08:17:13.100083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.100050 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-16457c0e-openshift-default-dockercfg-x2flj\"" Nov 02 08:17:13.121548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.121518 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:13.138811 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.138779 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-certs\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.138984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.138868 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-socket\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.139047 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.138979 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-istio-token\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.139047 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.139035 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50451fd9-2f4e-4871-ac49-b049890a1329-istiod-ca-cert\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.139169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.139138 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjm2j\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-kube-api-access-jjm2j\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.139323 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.139305 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-credential-socket\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.139396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.139354 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-envoy\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.139396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.139383 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50451fd9-2f4e-4871-ac49-b049890a1329-istio-podinfo\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.139491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.139471 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-data\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.224608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.224570 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:13.240916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.240885 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-istio-token\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241036 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.240950 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50451fd9-2f4e-4871-ac49-b049890a1329-istiod-ca-cert\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241036 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.240978 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjm2j\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-kube-api-access-jjm2j\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241036 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241030 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-credential-socket\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241279 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241059 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-envoy\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241279 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241079 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50451fd9-2f4e-4871-ac49-b049890a1329-istio-podinfo\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241422 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241395 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-data\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241506 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241484 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-credential-socket\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241507 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-certs\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241538 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-socket\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241767 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50451fd9-2f4e-4871-ac49-b049890a1329-istiod-ca-cert\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.241890 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241845 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-certs\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.242110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241890 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-socket\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.242110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.241897 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-data\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.243766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.243739 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50451fd9-2f4e-4871-ac49-b049890a1329-istio-podinfo\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.243849 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.243817 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-envoy\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.297521 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.297270 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-istio-token\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.297816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.297600 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjm2j\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-kube-api-access-jjm2j\") pod \"e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.315574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.315492 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:13.350688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.350648 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:13.592622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.592602 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs"] Nov 02 08:17:13.594737 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:13.594715 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f96b945_6b30_4cf3_82d4_4a8118488cd7.slice/crio-4b56199462f974f00c642f3ff78574fde41ded27b15f87e9b2b5254b40f888dc WatchSource:0}: Error finding container 4b56199462f974f00c642f3ff78574fde41ded27b15f87e9b2b5254b40f888dc: Status 404 returned error can't find the container with id 4b56199462f974f00c642f3ff78574fde41ded27b15f87e9b2b5254b40f888dc Nov 02 08:17:13.703765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.703743 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2"] Nov 02 08:17:13.706225 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:13.706159 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50451fd9_2f4e_4871_ac49_b049890a1329.slice/crio-8a00255bc5a625fd6ecbc4ecd57d426cda350237ef4650fa27cf27111683b87a WatchSource:0}: Error finding container 8a00255bc5a625fd6ecbc4ecd57d426cda350237ef4650fa27cf27111683b87a: Status 404 returned error can't find the container with id 8a00255bc5a625fd6ecbc4ecd57d426cda350237ef4650fa27cf27111683b87a Nov 02 08:17:13.708683 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.708638 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:17:13.708773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.708731 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:17:13.708811 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:13.708778 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:17:14.034266 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.034228 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:14.061433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.061363 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" event={"ID":"50451fd9-2f4e-4871-ac49-b049890a1329","Type":"ContainerStarted","Data":"0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10"} Nov 02 08:17:14.061601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.061439 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" event={"ID":"50451fd9-2f4e-4871-ac49-b049890a1329","Type":"ContainerStarted","Data":"8a00255bc5a625fd6ecbc4ecd57d426cda350237ef4650fa27cf27111683b87a"} Nov 02 08:17:14.063360 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.063326 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" event={"ID":"2f96b945-6b30-4cf3-82d4-4a8118488cd7","Type":"ContainerStarted","Data":"4b56199462f974f00c642f3ff78574fde41ded27b15f87e9b2b5254b40f888dc"} Nov 02 08:17:14.181964 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.181911 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" podStartSLOduration=2.181896862 podStartE2EDuration="2.181896862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:14.179640786 +0000 UTC m=+632.733145028" watchObservedRunningTime="2026-11-02 08:17:14.181896862 +0000 UTC m=+632.735401059" Nov 02 08:17:14.286943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.286857 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:14.316784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.316750 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:14.321926 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.321892 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:14.477692 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.477653 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:14.694516 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.694476 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8"] Nov 02 08:17:14.699071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.699041 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:14.753931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.753897 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8"] Nov 02 08:17:14.764056 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.764028 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-09e3828d\"" Nov 02 08:17:14.764255 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.764201 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-09e3828d-dockercfg-5k76x\"" Nov 02 08:17:14.856341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.856259 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160f5aa-0a01-4a17-8ada-51528bbdcc40-plugins-config-volume\") pod \"payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8\" (UID: \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:14.856341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.856292 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pstt\" (UniqueName: \"kubernetes.io/projected/b160f5aa-0a01-4a17-8ada-51528bbdcc40-kube-api-access-7pstt\") pod \"payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8\" (UID: \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:14.957647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.957605 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160f5aa-0a01-4a17-8ada-51528bbdcc40-plugins-config-volume\") pod \"payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8\" (UID: \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:14.957647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.957652 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pstt\" (UniqueName: \"kubernetes.io/projected/b160f5aa-0a01-4a17-8ada-51528bbdcc40-kube-api-access-7pstt\") pod \"payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8\" (UID: \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:14.958361 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:14.958342 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160f5aa-0a01-4a17-8ada-51528bbdcc40-plugins-config-volume\") pod \"payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8\" (UID: \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:15.012361 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.012327 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pstt\" (UniqueName: \"kubernetes.io/projected/b160f5aa-0a01-4a17-8ada-51528bbdcc40-kube-api-access-7pstt\") pod \"payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8\" (UID: \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\") " pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:15.019191 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.019142 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:15.020394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.020364 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:15.072735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.072067 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" event={"ID":"2f96b945-6b30-4cf3-82d4-4a8118488cd7","Type":"ContainerStarted","Data":"89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1"} Nov 02 08:17:15.072735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.072469 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:15.073026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.073003 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:15.073678 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.073651 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:17:15.184746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.184700 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" podStartSLOduration=2.219781351 podStartE2EDuration="3.184686759s" totalImagesPullingTime="964.905408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:15.182020479 +0000 UTC m=+633.735524676" watchObservedRunningTime="2026-11-02 08:17:15.184686759 +0000 UTC m=+633.738190955" Nov 02 08:17:15.207687 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.207642 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:15.389022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.388998 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8"] Nov 02 08:17:15.390886 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:15.390859 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb160f5aa_0a01_4a17_8ada_51528bbdcc40.slice/crio-652f71d7b18cc9b659c290e28dfbe80f53fb4b925d2ab3505374c219dcd2a59b WatchSource:0}: Error finding container 652f71d7b18cc9b659c290e28dfbe80f53fb4b925d2ab3505374c219dcd2a59b: Status 404 returned error can't find the container with id 652f71d7b18cc9b659c290e28dfbe80f53fb4b925d2ab3505374c219dcd2a59b Nov 02 08:17:15.394101 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.394073 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:15.596493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:15.596455 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:16.080770 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:16.080385 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" event={"ID":"b160f5aa-0a01-4a17-8ada-51528bbdcc40","Type":"ContainerStarted","Data":"4363f7734279963877a20cbc71a053edd5754ff598cb25d9b89278a91e76637e"} Nov 02 08:17:16.080770 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:16.080441 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" event={"ID":"b160f5aa-0a01-4a17-8ada-51528bbdcc40","Type":"ContainerStarted","Data":"652f71d7b18cc9b659c290e28dfbe80f53fb4b925d2ab3505374c219dcd2a59b"} Nov 02 08:17:16.081039 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:16.080935 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:16.198219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:16.198144 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" podStartSLOduration=2.198124608 podStartE2EDuration="2.198124608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:16.196470624 +0000 UTC m=+634.749974821" watchObservedRunningTime="2026-11-02 08:17:16.198124608 +0000 UTC m=+634.751628805" Nov 02 08:17:16.823530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:16.823495 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:16.981622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:16.981582 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:17.185393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.185362 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:17.228595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.228570 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb"] Nov 02 08:17:17.232283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.232266 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:17.283563 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.283539 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-53cc8e62\"" Nov 02 08:17:17.283563 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.283542 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-53cc8e62-dockercfg-9r2v4\"" Nov 02 08:17:17.331845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.331799 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb"] Nov 02 08:17:17.380609 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.380565 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb\" (UID: \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:17.380802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.380623 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmvn\" (UniqueName: \"kubernetes.io/projected/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-kube-api-access-2kmvn\") pod \"payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb\" (UID: \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:17.383279 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.383250 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb"] Nov 02 08:17:17.386964 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.386943 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:17.442881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.442800 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb"] Nov 02 08:17:17.481973 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.481940 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8xr\" (UniqueName: \"kubernetes.io/projected/bad6b514-7e68-475f-a3e2-5326aae05f55-kube-api-access-7p8xr\") pod \"payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb\" (UID: \"bad6b514-7e68-475f-a3e2-5326aae05f55\") " pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:17.482158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.481981 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb\" (UID: \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:17.482158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.482017 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmvn\" (UniqueName: \"kubernetes.io/projected/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-kube-api-access-2kmvn\") pod \"payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb\" (UID: \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:17.482158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.482108 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad6b514-7e68-475f-a3e2-5326aae05f55-plugins-config-volume\") pod \"payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb\" (UID: \"bad6b514-7e68-475f-a3e2-5326aae05f55\") " pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:17.482684 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.482667 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb\" (UID: \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:17.542436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.542402 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmvn\" (UniqueName: \"kubernetes.io/projected/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-kube-api-access-2kmvn\") pod \"payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb\" (UID: \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:17.547148 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.547123 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:17.583033 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.582998 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8xr\" (UniqueName: \"kubernetes.io/projected/bad6b514-7e68-475f-a3e2-5326aae05f55-kube-api-access-7p8xr\") pod \"payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb\" (UID: \"bad6b514-7e68-475f-a3e2-5326aae05f55\") " pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:17.583461 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.583088 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad6b514-7e68-475f-a3e2-5326aae05f55-plugins-config-volume\") pod \"payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb\" (UID: \"bad6b514-7e68-475f-a3e2-5326aae05f55\") " pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:17.583962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.583940 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad6b514-7e68-475f-a3e2-5326aae05f55-plugins-config-volume\") pod \"payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb\" (UID: \"bad6b514-7e68-475f-a3e2-5326aae05f55\") " pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:17.640478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.640442 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8xr\" (UniqueName: \"kubernetes.io/projected/bad6b514-7e68-475f-a3e2-5326aae05f55-kube-api-access-7p8xr\") pod \"payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb\" (UID: \"bad6b514-7e68-475f-a3e2-5326aae05f55\") " pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:17.700068 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.699982 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:17.920682 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.920657 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb"] Nov 02 08:17:17.921936 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:17.921908 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1466642_6fe1_40b6_8346_eb2e95a1e9e3.slice/crio-c3a38e4b563a6b4c4a0bd9c9db8762efdf3932414154a60b033d80db8bb66db4 WatchSource:0}: Error finding container c3a38e4b563a6b4c4a0bd9c9db8762efdf3932414154a60b033d80db8bb66db4: Status 404 returned error can't find the container with id c3a38e4b563a6b4c4a0bd9c9db8762efdf3932414154a60b033d80db8bb66db4 Nov 02 08:17:17.944721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:17.944697 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:18.071323 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:18.071296 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb"] Nov 02 08:17:18.074249 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:18.074212 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad6b514_7e68_475f_a3e2_5326aae05f55.slice/crio-52d57f8ef8dfa9557773734b382593bed4da9649774afa04ace857266f62d954 WatchSource:0}: Error finding container 52d57f8ef8dfa9557773734b382593bed4da9649774afa04ace857266f62d954: Status 404 returned error can't find the container with id 52d57f8ef8dfa9557773734b382593bed4da9649774afa04ace857266f62d954 Nov 02 08:17:18.091028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:18.091003 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" event={"ID":"bad6b514-7e68-475f-a3e2-5326aae05f55","Type":"ContainerStarted","Data":"52d57f8ef8dfa9557773734b382593bed4da9649774afa04ace857266f62d954"} Nov 02 08:17:18.092813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:18.092771 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" event={"ID":"b1466642-6fe1-40b6-8346-eb2e95a1e9e3","Type":"ContainerStarted","Data":"7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6"} Nov 02 08:17:18.092937 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:18.092849 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" event={"ID":"b1466642-6fe1-40b6-8346-eb2e95a1e9e3","Type":"ContainerStarted","Data":"c3a38e4b563a6b4c4a0bd9c9db8762efdf3932414154a60b033d80db8bb66db4"} Nov 02 08:17:18.116121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:18.116089 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:18.204376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:18.204322 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" podStartSLOduration=1.204308216 podStartE2EDuration="1.204308216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:18.202132141 +0000 UTC m=+636.755636358" watchObservedRunningTime="2026-11-02 08:17:18.204308216 +0000 UTC m=+636.757812392" Nov 02 08:17:18.277445 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:18.277362 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:19.099644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:19.099606 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" event={"ID":"bad6b514-7e68-475f-a3e2-5326aae05f55","Type":"ContainerStarted","Data":"b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d"} Nov 02 08:17:19.100103 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:19.099852 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:19.100103 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:19.099880 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:19.209644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:19.209589 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" podStartSLOduration=2.209575637 podStartE2EDuration="2.209575637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:19.208478594 +0000 UTC m=+637.761982792" watchObservedRunningTime="2026-11-02 08:17:19.209575637 +0000 UTC m=+637.763079833" Nov 02 08:17:19.427170 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:19.427123 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:19.600394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:19.600336 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:21.724723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:21.724623 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:21.885749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:21.885715 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:22.051524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:22.051442 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:17:22.051524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:22.051523 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:17:22.089762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:22.089736 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:17:23.419247 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:23.419210 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:23.755592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:23.755500 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:23.935283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:23.935242 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:24.468911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:24.468877 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:24.695400 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:24.695365 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:24.857866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:24.857777 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw"] Nov 02 08:17:24.862842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:24.862817 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:24.958613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:24.958570 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f7h\" (UniqueName: \"kubernetes.io/projected/84488f2a-6ede-40e9-a394-2db0a6f25669-kube-api-access-r4f7h\") pod \"payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw\" (UID: \"84488f2a-6ede-40e9-a394-2db0a6f25669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:24.958613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:24.958608 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84488f2a-6ede-40e9-a394-2db0a6f25669-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw\" (UID: \"84488f2a-6ede-40e9-a394-2db0a6f25669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:24.958886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:24.958785 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d0628e\"" Nov 02 08:17:24.958886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:24.958785 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d0628e-dockercfg-vdlgn\"" Nov 02 08:17:24.961986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:24.961959 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw"] Nov 02 08:17:25.059412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:25.059375 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f7h\" (UniqueName: \"kubernetes.io/projected/84488f2a-6ede-40e9-a394-2db0a6f25669-kube-api-access-r4f7h\") pod \"payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw\" (UID: \"84488f2a-6ede-40e9-a394-2db0a6f25669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:25.059412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:25.059418 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84488f2a-6ede-40e9-a394-2db0a6f25669-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw\" (UID: \"84488f2a-6ede-40e9-a394-2db0a6f25669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:25.060125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:25.060104 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84488f2a-6ede-40e9-a394-2db0a6f25669-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw\" (UID: \"84488f2a-6ede-40e9-a394-2db0a6f25669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:25.113096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:25.113019 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f7h\" (UniqueName: \"kubernetes.io/projected/84488f2a-6ede-40e9-a394-2db0a6f25669-kube-api-access-r4f7h\") pod \"payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw\" (UID: \"84488f2a-6ede-40e9-a394-2db0a6f25669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:25.175766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:25.175734 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:25.543038 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:25.542943 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw"] Nov 02 08:17:25.546405 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:25.546370 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84488f2a_6ede_40e9_a394_2db0a6f25669.slice/crio-d24e3c9f1ac98aac485f117f5f574d147bdbb47f0e71ecf07e6151e4a067f2be WatchSource:0}: Error finding container d24e3c9f1ac98aac485f117f5f574d147bdbb47f0e71ecf07e6151e4a067f2be: Status 404 returned error can't find the container with id d24e3c9f1ac98aac485f117f5f574d147bdbb47f0e71ecf07e6151e4a067f2be Nov 02 08:17:26.137508 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:26.137474 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" event={"ID":"84488f2a-6ede-40e9-a394-2db0a6f25669","Type":"ContainerStarted","Data":"da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75"} Nov 02 08:17:26.137687 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:26.137531 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" event={"ID":"84488f2a-6ede-40e9-a394-2db0a6f25669","Type":"ContainerStarted","Data":"d24e3c9f1ac98aac485f117f5f574d147bdbb47f0e71ecf07e6151e4a067f2be"} Nov 02 08:17:26.137687 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:26.137663 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:26.247054 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:26.247005 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" podStartSLOduration=2.246988287 podStartE2EDuration="2.246988287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:26.24502784 +0000 UTC m=+644.798532059" watchObservedRunningTime="2026-11-02 08:17:26.246988287 +0000 UTC m=+644.800492483" Nov 02 08:17:28.096120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:28.096091 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:17:28.620574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:28.620535 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:29.403398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.403366 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:29.577374 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.577332 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:29.792090 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.792054 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms"] Nov 02 08:17:29.795426 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.795410 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:29.804975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.804951 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj2hq\" (UniqueName: \"kubernetes.io/projected/33260340-d2c8-4ce3-8483-455a27450dbb-kube-api-access-kj2hq\") pod \"payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms\" (UID: \"33260340-d2c8-4ce3-8483-455a27450dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:29.805098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.805077 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33260340-d2c8-4ce3-8483-455a27450dbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms\" (UID: \"33260340-d2c8-4ce3-8483-455a27450dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:29.846377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.846355 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2e734c1c\"" Nov 02 08:17:29.893278 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.892846 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2e734c1c-dockercfg-zz8mc\"" Nov 02 08:17:29.897381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.897354 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms"] Nov 02 08:17:29.905802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.905774 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj2hq\" (UniqueName: \"kubernetes.io/projected/33260340-d2c8-4ce3-8483-455a27450dbb-kube-api-access-kj2hq\") pod \"payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms\" (UID: \"33260340-d2c8-4ce3-8483-455a27450dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:29.905953 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.905846 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33260340-d2c8-4ce3-8483-455a27450dbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms\" (UID: \"33260340-d2c8-4ce3-8483-455a27450dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:29.906580 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.906559 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33260340-d2c8-4ce3-8483-455a27450dbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms\" (UID: \"33260340-d2c8-4ce3-8483-455a27450dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:29.959393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:29.959362 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj2hq\" (UniqueName: \"kubernetes.io/projected/33260340-d2c8-4ce3-8483-455a27450dbb-kube-api-access-kj2hq\") pod \"payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms\" (UID: \"33260340-d2c8-4ce3-8483-455a27450dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:30.107512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:30.107431 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:17:30.107512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:30.107495 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:17:30.107860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:30.107837 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:30.472599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:30.472569 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms"] Nov 02 08:17:30.474579 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:30.474548 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33260340_d2c8_4ce3_8483_455a27450dbb.slice/crio-2ddf1eca8d38ba72137273f4d6726f59887db14a382990640dd4ed75126ef96b WatchSource:0}: Error finding container 2ddf1eca8d38ba72137273f4d6726f59887db14a382990640dd4ed75126ef96b: Status 404 returned error can't find the container with id 2ddf1eca8d38ba72137273f4d6726f59887db14a382990640dd4ed75126ef96b Nov 02 08:17:31.168689 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:31.168655 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" event={"ID":"33260340-d2c8-4ce3-8483-455a27450dbb","Type":"ContainerStarted","Data":"f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520"} Nov 02 08:17:31.168869 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:31.168704 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" event={"ID":"33260340-d2c8-4ce3-8483-455a27450dbb","Type":"ContainerStarted","Data":"2ddf1eca8d38ba72137273f4d6726f59887db14a382990640dd4ed75126ef96b"} Nov 02 08:17:31.168869 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:31.168774 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:31.276686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:31.276621 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" podStartSLOduration=2.27660745 podStartE2EDuration="2.27660745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:31.274347355 +0000 UTC m=+649.827851552" watchObservedRunningTime="2026-11-02 08:17:31.27660745 +0000 UTC m=+649.830111647" Nov 02 08:17:32.006967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.006934 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:32.213359 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.213319 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:32.546492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.546451 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p"] Nov 02 08:17:32.550412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.550394 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:32.603909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.603871 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p"] Nov 02 08:17:32.630572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.630527 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-plugins-config-volume\") pod \"payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p\" (UID: \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\") " pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:32.630748 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.630679 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgdkr\" (UniqueName: \"kubernetes.io/projected/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-kube-api-access-qgdkr\") pod \"payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p\" (UID: \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\") " pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:32.646557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.646526 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-16457c0e\"" Nov 02 08:17:32.646821 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.646802 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-16457c0e-dockercfg-xrz5b\"" Nov 02 08:17:32.731494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.731462 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgdkr\" (UniqueName: \"kubernetes.io/projected/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-kube-api-access-qgdkr\") pod \"payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p\" (UID: \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\") " pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:32.731649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.731561 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-plugins-config-volume\") pod \"payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p\" (UID: \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\") " pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:32.732311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.732291 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-plugins-config-volume\") pod \"payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p\" (UID: \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\") " pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:32.785293 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.785262 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgdkr\" (UniqueName: \"kubernetes.io/projected/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-kube-api-access-qgdkr\") pod \"payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p\" (UID: \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\") " pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:32.863400 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:32.863310 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:33.235395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:33.235363 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p"] Nov 02 08:17:33.236733 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:33.236710 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bae3ef5_b8b8_4cc8_bb22_5796c7242d7a.slice/crio-ccc1b76eaccbcbfdd2072d9258c91ce11e13a0eccdc972b0b6c3e467b642a8f3 WatchSource:0}: Error finding container ccc1b76eaccbcbfdd2072d9258c91ce11e13a0eccdc972b0b6c3e467b642a8f3: Status 404 returned error can't find the container with id ccc1b76eaccbcbfdd2072d9258c91ce11e13a0eccdc972b0b6c3e467b642a8f3 Nov 02 08:17:33.751191 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:33.751142 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:34.187164 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:34.187123 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" event={"ID":"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a","Type":"ContainerStarted","Data":"8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f"} Nov 02 08:17:34.187345 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:34.187207 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" event={"ID":"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a","Type":"ContainerStarted","Data":"ccc1b76eaccbcbfdd2072d9258c91ce11e13a0eccdc972b0b6c3e467b642a8f3"} Nov 02 08:17:34.187345 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:34.187237 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:34.295954 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:34.295901 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" podStartSLOduration=2.295883097 podStartE2EDuration="2.295883097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:34.294717233 +0000 UTC m=+652.848221430" watchObservedRunningTime="2026-11-02 08:17:34.295883097 +0000 UTC m=+652.849387294" Nov 02 08:17:37.083072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.083039 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:37.146738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.146713 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:17:37.295014 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.294976 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:37.477616 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.477586 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh"] Nov 02 08:17:37.481423 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.481407 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:37.532339 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.532311 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e6ca80-dockercfg-pnmpg\"" Nov 02 08:17:37.532555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.532306 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e6ca80\"" Nov 02 08:17:37.536055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.536025 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh"] Nov 02 08:17:37.578553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.578515 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxhxd\" (UniqueName: \"kubernetes.io/projected/466de682-78b8-4b8f-8e6e-6fd1205f41b2-kube-api-access-pxhxd\") pod \"payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh\" (UID: \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:37.578724 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.578685 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466de682-78b8-4b8f-8e6e-6fd1205f41b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh\" (UID: \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:37.680055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.680017 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxhxd\" (UniqueName: \"kubernetes.io/projected/466de682-78b8-4b8f-8e6e-6fd1205f41b2-kube-api-access-pxhxd\") pod \"payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh\" (UID: \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:37.680268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.680252 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466de682-78b8-4b8f-8e6e-6fd1205f41b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh\" (UID: \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:37.681241 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.681215 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466de682-78b8-4b8f-8e6e-6fd1205f41b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh\" (UID: \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:37.737222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.737124 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxhxd\" (UniqueName: \"kubernetes.io/projected/466de682-78b8-4b8f-8e6e-6fd1205f41b2-kube-api-access-pxhxd\") pod \"payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh\" (UID: \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:37.794062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:37.794015 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:38.194612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:38.194584 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh"] Nov 02 08:17:38.196644 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:38.196616 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466de682_78b8_4b8f_8e6e_6fd1205f41b2.slice/crio-3a739ccce868266bc5919da5aedc1f81578ed1925800675631fb778e32e0fdcb WatchSource:0}: Error finding container 3a739ccce868266bc5919da5aedc1f81578ed1925800675631fb778e32e0fdcb: Status 404 returned error can't find the container with id 3a739ccce868266bc5919da5aedc1f81578ed1925800675631fb778e32e0fdcb Nov 02 08:17:38.236477 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:38.236321 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" event={"ID":"466de682-78b8-4b8f-8e6e-6fd1205f41b2","Type":"ContainerStarted","Data":"3a739ccce868266bc5919da5aedc1f81578ed1925800675631fb778e32e0fdcb"} Nov 02 08:17:38.570380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:38.570272 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:38.775535 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:38.775495 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:38.934993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:38.934912 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:39.244046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:39.243959 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" event={"ID":"466de682-78b8-4b8f-8e6e-6fd1205f41b2","Type":"ContainerStarted","Data":"b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84"} Nov 02 08:17:39.244440 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:39.244099 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:39.351137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:39.351075 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" podStartSLOduration=2.351059758 podStartE2EDuration="2.351059758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:39.349550668 +0000 UTC m=+657.903054844" watchObservedRunningTime="2026-11-02 08:17:39.351059758 +0000 UTC m=+657.904563999" Nov 02 08:17:40.521207 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:17:40.521149 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:17:42.178218 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:42.178190 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:17:43.612506 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.612473 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z"] Nov 02 08:17:43.615371 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.615355 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.672897 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.672867 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z"] Nov 02 08:17:43.691256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.691225 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2e734c1c-serving-cert\"" Nov 02 08:17:43.736283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.736243 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkp7b\" (UniqueName: \"kubernetes.io/projected/3ff6956f-c65a-4c0a-91ab-30a185c84019-kube-api-access-vkp7b\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.736441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.736296 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.736441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.736375 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-metrics-certs\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.736441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.736434 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff6956f-c65a-4c0a-91ab-30a185c84019-openshift-service-ca\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.837148 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.837070 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff6956f-c65a-4c0a-91ab-30a185c84019-openshift-service-ca\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.837313 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.837199 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkp7b\" (UniqueName: \"kubernetes.io/projected/3ff6956f-c65a-4c0a-91ab-30a185c84019-kube-api-access-vkp7b\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.837313 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.837244 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.837313 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.837280 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-metrics-certs\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.837427 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:17:43.837409 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-serving-cert: secret "maas-api-e2e-ait-2e734c1c-serving-cert" not found Nov 02 08:17:43.837490 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:17:43.837479 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls podName:3ff6956f-c65a-4c0a-91ab-30a185c84019 nodeName:}" failed. No retries permitted until 2026-11-02 08:17:44.337464554 +0000 UTC m=+662.890968730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls") pod "maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" (UID: "3ff6956f-c65a-4c0a-91ab-30a185c84019") : secret "maas-api-e2e-ait-2e734c1c-serving-cert" not found Nov 02 08:17:43.837854 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.837835 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff6956f-c65a-4c0a-91ab-30a185c84019-openshift-service-ca\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.839821 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.839801 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-metrics-certs\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.897993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.897963 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkp7b\" (UniqueName: \"kubernetes.io/projected/3ff6956f-c65a-4c0a-91ab-30a185c84019-kube-api-access-vkp7b\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:43.901077 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:43.901046 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:44.129839 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.129754 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:44.273948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.273911 6160 generic.go:356] "Generic (PLEG): container finished" podID="c3242831-c651-4f6b-953f-32726b062cc4" containerID="ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda" exitCode=0 Nov 02 08:17:44.274134 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.273982 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" event={"ID":"c3242831-c651-4f6b-953f-32726b062cc4","Type":"ContainerDied","Data":"ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda"} Nov 02 08:17:44.274387 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.274373 6160 scope.go:122] "RemoveContainer" containerID="ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda" Nov 02 08:17:44.275655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.275619 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:17:44.275774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.275707 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:17:44.275774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.275757 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:17:44.306029 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.305993 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:44.343603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.343578 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:44.345913 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.345890 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls\") pod \"maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:44.527399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.527365 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:44.900591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:44.900566 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z"] Nov 02 08:17:44.902149 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:44.902121 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff6956f_c65a_4c0a_91ab_30a185c84019.slice/crio-53b72e76c38b2f67e0d275a2da81f5d276c1c1818e9d4acd13112b0a905c9818 WatchSource:0}: Error finding container 53b72e76c38b2f67e0d275a2da81f5d276c1c1818e9d4acd13112b0a905c9818: Status 404 returned error can't find the container with id 53b72e76c38b2f67e0d275a2da81f5d276c1c1818e9d4acd13112b0a905c9818 Nov 02 08:17:45.196221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:45.196164 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:17:45.282059 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:45.282025 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" event={"ID":"c3242831-c651-4f6b-953f-32726b062cc4","Type":"ContainerStarted","Data":"fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b"} Nov 02 08:17:45.283347 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:45.283321 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" event={"ID":"3ff6956f-c65a-4c0a-91ab-30a185c84019","Type":"ContainerStarted","Data":"53b72e76c38b2f67e0d275a2da81f5d276c1c1818e9d4acd13112b0a905c9818"} Nov 02 08:17:45.990364 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:45.990328 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:45.990828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:45.990806 6160 patch_prober.go:29] interesting pod/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 08:17:45.990881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:45.990866 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 08:17:46.290656 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:46.290620 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" event={"ID":"3ff6956f-c65a-4c0a-91ab-30a185c84019","Type":"ContainerStarted","Data":"d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db"} Nov 02 08:17:46.290837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:46.290722 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:46.401646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:46.401595 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" podStartSLOduration=2.45974907 podStartE2EDuration="3.401576994s" totalImagesPullingTime="941.827924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:46.399890215 +0000 UTC m=+664.953394411" watchObservedRunningTime="2026-11-02 08:17:46.401576994 +0000 UTC m=+664.955081190" Nov 02 08:17:46.990111 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:46.990078 6160 patch_prober.go:29] interesting pod/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 08:17:46.990306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:46.990130 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 08:17:47.990577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:47.990536 6160 patch_prober.go:29] interesting pod/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 08:17:47.991058 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:47.990595 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 08:17:48.990853 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:48.990817 6160 patch_prober.go:29] interesting pod/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 08:17:48.991311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:48.990881 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 08:17:49.289787 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:49.289697 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:49.489610 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:49.489569 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:49.743578 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:49.743533 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:49.990843 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:49.990808 6160 patch_prober.go:29] interesting pod/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 08:17:49.991016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:49.990859 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 08:17:50.253374 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:50.253341 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:17:50.995055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:50.995026 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:50.995555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:50.995442 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:50.996229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:50.996210 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:17:52.301115 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:52.301085 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:17:54.070163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.070127 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv"] Nov 02 08:17:54.073790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.073772 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.122373 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.122349 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-e6ca80-serving-cert\"" Nov 02 08:17:54.146048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.146018 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b219eb-aa01-480c-b92b-8ce43fff3718-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.146243 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.146071 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-metrics-certs\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.146243 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.146226 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.146365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.146273 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxwb\" (UniqueName: \"kubernetes.io/projected/f4b219eb-aa01-480c-b92b-8ce43fff3718-kube-api-access-bmxwb\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.171255 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.171225 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv"] Nov 02 08:17:54.247463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.247427 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-metrics-certs\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.247661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.247488 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.247661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.247510 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxwb\" (UniqueName: \"kubernetes.io/projected/f4b219eb-aa01-480c-b92b-8ce43fff3718-kube-api-access-bmxwb\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.247661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.247586 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b219eb-aa01-480c-b92b-8ce43fff3718-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.247661 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:17:54.247601 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-serving-cert: secret "maas-api-e2e-worker-w0-e6ca80-serving-cert" not found Nov 02 08:17:54.247880 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:17:54.247675 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls podName:f4b219eb-aa01-480c-b92b-8ce43fff3718 nodeName:}" failed. No retries permitted until 2026-11-02 08:17:54.74765488 +0000 UTC m=+673.301159058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls") pod "maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" (UID: "f4b219eb-aa01-480c-b92b-8ce43fff3718") : secret "maas-api-e2e-worker-w0-e6ca80-serving-cert" not found Nov 02 08:17:54.248262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.248245 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b219eb-aa01-480c-b92b-8ce43fff3718-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.250154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.250123 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-metrics-certs\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.284506 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.284465 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:54.301635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.301611 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxwb\" (UniqueName: \"kubernetes.io/projected/f4b219eb-aa01-480c-b92b-8ce43fff3718-kube-api-access-bmxwb\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.485092 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.485031 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:54.662067 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.662021 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:54.752523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.752434 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.755148 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.755121 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:54.987466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:54.987426 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:55.358016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:55.357990 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv"] Nov 02 08:17:55.359719 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:17:55.359691 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b219eb_aa01_480c_b92b_8ce43fff3718.slice/crio-cd2482b614431a2eed7a4922d50062d896098b2744f1879abc9808385ab7b844 WatchSource:0}: Error finding container cd2482b614431a2eed7a4922d50062d896098b2744f1879abc9808385ab7b844: Status 404 returned error can't find the container with id cd2482b614431a2eed7a4922d50062d896098b2744f1879abc9808385ab7b844 Nov 02 08:17:55.361198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:55.361163 6160 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:17:56.348654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:56.348613 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" event={"ID":"f4b219eb-aa01-480c-b92b-8ce43fff3718","Type":"ContainerStarted","Data":"0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335"} Nov 02 08:17:56.348809 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:56.348792 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:17:56.348852 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:56.348822 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" event={"ID":"f4b219eb-aa01-480c-b92b-8ce43fff3718","Type":"ContainerStarted","Data":"cd2482b614431a2eed7a4922d50062d896098b2744f1879abc9808385ab7b844"} Nov 02 08:17:56.500002 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:56.499901 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" podStartSLOduration=1.625743644 podStartE2EDuration="2.499887261s" totalImagesPullingTime="874.143617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:56.497528672 +0000 UTC m=+675.051032870" watchObservedRunningTime="2026-11-02 08:17:56.499887261 +0000 UTC m=+675.053391457" Nov 02 08:17:59.348142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:17:59.348111 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:01.485657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:01.485616 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:02.360891 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:02.360862 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:18:04.144528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:04.144487 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:06.203938 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:06.203903 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8"] Nov 02 08:18:06.411842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:06.411805 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:06.601730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:06.601638 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:06.755673 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:06.755638 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:07.414985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:07.414947 6160 generic.go:356] "Generic (PLEG): container finished" podID="50451fd9-2f4e-4871-ac49-b049890a1329" containerID="0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10" exitCode=0 Nov 02 08:18:07.415401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:07.415019 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" event={"ID":"50451fd9-2f4e-4871-ac49-b049890a1329","Type":"ContainerDied","Data":"0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10"} Nov 02 08:18:07.415401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:07.415392 6160 scope.go:122] "RemoveContainer" containerID="0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10" Nov 02 08:18:07.416658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:07.416627 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:18:07.416754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:07.416695 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:18:07.416754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:07.416727 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:18:08.440108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:08.440070 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" event={"ID":"50451fd9-2f4e-4871-ac49-b049890a1329","Type":"ContainerStarted","Data":"e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545"} Nov 02 08:18:08.964528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:08.964483 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:09.326258 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:09.322594 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:18:09.326258 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:09.322723 6160 patch_prober.go:29] interesting pod/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 08:18:09.326258 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:09.322764 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 08:18:10.317690 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:10.317648 6160 patch_prober.go:29] interesting pod/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 08:18:10.318347 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:10.318316 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 08:18:10.543164 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:10.543110 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:18:11.204328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.204285 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" podUID="b160f5aa-0a01-4a17-8ada-51528bbdcc40" containerName="payload-processing" containerID="cri-o://4363f7734279963877a20cbc71a053edd5754ff598cb25d9b89278a91e76637e" gracePeriod=25 Nov 02 08:18:11.327200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.326724 6160 patch_prober.go:29] interesting pod/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 08:18:11.327200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.326796 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 08:18:11.472522 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.472433 6160 generic.go:356] "Generic (PLEG): container finished" podID="b160f5aa-0a01-4a17-8ada-51528bbdcc40" containerID="4363f7734279963877a20cbc71a053edd5754ff598cb25d9b89278a91e76637e" exitCode=0 Nov 02 08:18:11.472657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.472505 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" event={"ID":"b160f5aa-0a01-4a17-8ada-51528bbdcc40","Type":"ContainerDied","Data":"4363f7734279963877a20cbc71a053edd5754ff598cb25d9b89278a91e76637e"} Nov 02 08:18:11.498329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.498306 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:18:11.559007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.558968 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b160f5aa-0a01-4a17-8ada-51528bbdcc40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160f5aa-0a01-4a17-8ada-51528bbdcc40-plugins-config-volume\") pod \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\" (UID: \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\") " Nov 02 08:18:11.559239 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.559131 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b160f5aa-0a01-4a17-8ada-51528bbdcc40-kube-api-access-7pstt\" (UniqueName: \"kubernetes.io/projected/b160f5aa-0a01-4a17-8ada-51528bbdcc40-kube-api-access-7pstt\") pod \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\" (UID: \"b160f5aa-0a01-4a17-8ada-51528bbdcc40\") " Nov 02 08:18:11.559459 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.559428 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b160f5aa-0a01-4a17-8ada-51528bbdcc40-plugins-config-volume" pod "b160f5aa-0a01-4a17-8ada-51528bbdcc40" (UID: "b160f5aa-0a01-4a17-8ada-51528bbdcc40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:11.561463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.561439 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b160f5aa-0a01-4a17-8ada-51528bbdcc40-kube-api-access-7pstt" pod "b160f5aa-0a01-4a17-8ada-51528bbdcc40" (UID: "b160f5aa-0a01-4a17-8ada-51528bbdcc40"). InnerVolumeSpecName "kube-api-access-7pstt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:11.660066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.660035 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pstt\" (UniqueName: \"kubernetes.io/projected/b160f5aa-0a01-4a17-8ada-51528bbdcc40-kube-api-access-7pstt\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:11.660066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:11.660065 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b160f5aa-0a01-4a17-8ada-51528bbdcc40-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:12.323065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:12.322581 6160 patch_prober.go:29] interesting pod/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 08:18:12.323065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:12.322662 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 08:18:12.479319 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:12.479284 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" Nov 02 08:18:12.479784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:12.479314 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8" event={"ID":"b160f5aa-0a01-4a17-8ada-51528bbdcc40","Type":"ContainerDied","Data":"652f71d7b18cc9b659c290e28dfbe80f53fb4b925d2ab3505374c219dcd2a59b"} Nov 02 08:18:12.479784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:12.479388 6160 scope.go:122] "RemoveContainer" containerID="4363f7734279963877a20cbc71a053edd5754ff598cb25d9b89278a91e76637e" Nov 02 08:18:12.637243 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:12.637142 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8"] Nov 02 08:18:12.683762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:12.683728 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09e3828d-648d8764c7-2smh8"] Nov 02 08:18:13.326462 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.326417 6160 patch_prober.go:29] interesting pod/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 08:18:13.326661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.326506 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 08:18:13.408863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.408821 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs"] Nov 02 08:18:13.409133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.409107 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" podUID="2f96b945-6b30-4cf3-82d4-4a8118488cd7" containerName="maas-api" containerID="cri-o://89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1" gracePeriod=30 Nov 02 08:18:13.539746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.539716 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:13.586556 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.584419 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b160f5aa-0a01-4a17-8ada-51528bbdcc40" path="/var/lib/kubelet/pods/b160f5aa-0a01-4a17-8ada-51528bbdcc40/volumes" Nov 02 08:18:13.669422 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.669394 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:18:13.680462 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.680440 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-metrics-certs\") pod \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " Nov 02 08:18:13.680611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.680491 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls\") pod \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " Nov 02 08:18:13.680611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.680557 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f96b945-6b30-4cf3-82d4-4a8118488cd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f96b945-6b30-4cf3-82d4-4a8118488cd7-openshift-service-ca\") pod \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " Nov 02 08:18:13.680719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.680657 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f96b945-6b30-4cf3-82d4-4a8118488cd7-kube-api-access-vhd9z\" (UniqueName: \"kubernetes.io/projected/2f96b945-6b30-4cf3-82d4-4a8118488cd7-kube-api-access-vhd9z\") pod \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\" (UID: \"2f96b945-6b30-4cf3-82d4-4a8118488cd7\") " Nov 02 08:18:13.680975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.680949 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f96b945-6b30-4cf3-82d4-4a8118488cd7-openshift-service-ca" pod "2f96b945-6b30-4cf3-82d4-4a8118488cd7" (UID: "2f96b945-6b30-4cf3-82d4-4a8118488cd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:13.682711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.682685 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-metrics-certs" pod "2f96b945-6b30-4cf3-82d4-4a8118488cd7" (UID: "2f96b945-6b30-4cf3-82d4-4a8118488cd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:18:13.682848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.682822 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls" pod "2f96b945-6b30-4cf3-82d4-4a8118488cd7" (UID: "2f96b945-6b30-4cf3-82d4-4a8118488cd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:18:13.682919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.682886 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f96b945-6b30-4cf3-82d4-4a8118488cd7-kube-api-access-vhd9z" pod "2f96b945-6b30-4cf3-82d4-4a8118488cd7" (UID: "2f96b945-6b30-4cf3-82d4-4a8118488cd7"). InnerVolumeSpecName "kube-api-access-vhd9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:13.782325 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.782292 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhd9z\" (UniqueName: \"kubernetes.io/projected/2f96b945-6b30-4cf3-82d4-4a8118488cd7-kube-api-access-vhd9z\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:13.782325 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.782324 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:13.782534 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.782339 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f96b945-6b30-4cf3-82d4-4a8118488cd7-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:13.782534 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:13.782351 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f96b945-6b30-4cf3-82d4-4a8118488cd7-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:14.320328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.320297 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:18:14.320703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.320684 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:18:14.321294 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.321275 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:18:14.494502 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.494472 6160 generic.go:356] "Generic (PLEG): container finished" podID="2f96b945-6b30-4cf3-82d4-4a8118488cd7" containerID="89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1" exitCode=0 Nov 02 08:18:14.494703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.494519 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" Nov 02 08:18:14.494703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.494556 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" event={"ID":"2f96b945-6b30-4cf3-82d4-4a8118488cd7","Type":"ContainerDied","Data":"89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1"} Nov 02 08:18:14.494703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.494612 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs" event={"ID":"2f96b945-6b30-4cf3-82d4-4a8118488cd7","Type":"ContainerDied","Data":"4b56199462f974f00c642f3ff78574fde41ded27b15f87e9b2b5254b40f888dc"} Nov 02 08:18:14.494703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.494642 6160 scope.go:122] "RemoveContainer" containerID="89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1" Nov 02 08:18:14.512120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.512091 6160 scope.go:122] "RemoveContainer" containerID="89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1" Nov 02 08:18:14.512406 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:14.512389 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1\": container with ID starting with 89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1 not found: ID does not exist" containerID="89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1" Nov 02 08:18:14.512455 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.512422 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1"} err="failed to get container status \"89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1\": rpc error: code = NotFound desc = could not find container \"89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1\": container with ID starting with 89c2fd446805aaf50ccbc3987119d8a83c46d5640ea6db5a22006ff5648975b1 not found: ID does not exist" Nov 02 08:18:14.644966 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.644879 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs"] Nov 02 08:18:14.650453 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:14.650427 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09e3828d-5b889d7fbd-274rs"] Nov 02 08:18:15.253564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:15.253515 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:15.465378 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:15.465342 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:15.583584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:15.583509 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f96b945-6b30-4cf3-82d4-4a8118488cd7" path="/var/lib/kubelet/pods/2f96b945-6b30-4cf3-82d4-4a8118488cd7/volumes" Nov 02 08:18:15.630200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:15.630146 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:15.834811 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:15.834722 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:16.595650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:16.595615 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:17.183598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:17.183560 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:17.653943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:17.653849 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:19.206787 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:19.206751 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c"] Nov 02 08:18:19.321286 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:19.321251 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz"] Nov 02 08:18:19.611726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:19.611638 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:19.823328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:19.823290 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:20.663759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:20.663731 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:22.291642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:22.291599 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:22.466844 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:22.466806 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:22.632723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:22.632623 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:24.184494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.184455 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb"] Nov 02 08:18:24.207629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.207589 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" podUID="ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611" containerName="payload-processing" containerID="cri-o://14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c" gracePeriod=25 Nov 02 08:18:24.299048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.299014 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb"] Nov 02 08:18:24.321912 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.321877 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" podUID="c963df6f-c03e-407e-8d8d-355141c73c40" containerName="payload-pre-processing" containerID="cri-o://f5056837d28a2b892b10c0dedfe71fb6a37bd72fd607f3356d77ea7cf2352961" gracePeriod=25 Nov 02 08:18:24.471252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.471215 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:18:24.565749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.565715 6160 generic.go:356] "Generic (PLEG): container finished" podID="c963df6f-c03e-407e-8d8d-355141c73c40" containerID="f5056837d28a2b892b10c0dedfe71fb6a37bd72fd607f3356d77ea7cf2352961" exitCode=0 Nov 02 08:18:24.565864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.565781 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" event={"ID":"c963df6f-c03e-407e-8d8d-355141c73c40","Type":"ContainerDied","Data":"f5056837d28a2b892b10c0dedfe71fb6a37bd72fd607f3356d77ea7cf2352961"} Nov 02 08:18:24.567345 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.567323 6160 generic.go:356] "Generic (PLEG): container finished" podID="ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611" containerID="14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c" exitCode=0 Nov 02 08:18:24.567465 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.567351 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" event={"ID":"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611","Type":"ContainerDied","Data":"14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c"} Nov 02 08:18:24.567465 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.567378 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" Nov 02 08:18:24.567465 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.567407 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c" event={"ID":"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611","Type":"ContainerDied","Data":"1d0be286c137cb3bf7bddca0622d709333e7c4d30389c5562e21d3cf31e93ae5"} Nov 02 08:18:24.567465 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.567440 6160 scope.go:122] "RemoveContainer" containerID="14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c" Nov 02 08:18:24.581404 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.581382 6160 scope.go:122] "RemoveContainer" containerID="14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c" Nov 02 08:18:24.581697 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:24.581672 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c\": container with ID starting with 14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c not found: ID does not exist" containerID="14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c" Nov 02 08:18:24.581803 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.581711 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c"} err="failed to get container status \"14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c\": rpc error: code = NotFound desc = could not find container \"14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c\": container with ID starting with 14d62ab1d9ac4ff35a61a314101575d7da10b2c5434cadf5f0efc4458f83b82c not found: ID does not exist" Nov 02 08:18:24.584590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.584572 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:18:24.586747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.586639 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-plugins-config-volume\") pod \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\" (UID: \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\") " Nov 02 08:18:24.586747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.586679 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-kube-api-access-mkwr2\" (UniqueName: \"kubernetes.io/projected/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-kube-api-access-mkwr2\") pod \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\" (UID: \"ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611\") " Nov 02 08:18:24.587074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.587049 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-plugins-config-volume" pod "ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611" (UID: "ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:24.587526 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.587501 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:24.589585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.589564 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-kube-api-access-mkwr2" pod "ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611" (UID: "ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611"). InnerVolumeSpecName "kube-api-access-mkwr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:24.687821 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.687790 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c963df6f-c03e-407e-8d8d-355141c73c40-kube-api-access-stv98\" (UniqueName: \"kubernetes.io/projected/c963df6f-c03e-407e-8d8d-355141c73c40-kube-api-access-stv98\") pod \"c963df6f-c03e-407e-8d8d-355141c73c40\" (UID: \"c963df6f-c03e-407e-8d8d-355141c73c40\") " Nov 02 08:18:24.688028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.687857 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c963df6f-c03e-407e-8d8d-355141c73c40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963df6f-c03e-407e-8d8d-355141c73c40-plugins-config-volume\") pod \"c963df6f-c03e-407e-8d8d-355141c73c40\" (UID: \"c963df6f-c03e-407e-8d8d-355141c73c40\") " Nov 02 08:18:24.688099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.688070 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:24.688099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.688082 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkwr2\" (UniqueName: \"kubernetes.io/projected/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611-kube-api-access-mkwr2\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:24.688244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.688190 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c963df6f-c03e-407e-8d8d-355141c73c40-plugins-config-volume" pod "c963df6f-c03e-407e-8d8d-355141c73c40" (UID: "c963df6f-c03e-407e-8d8d-355141c73c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:24.690101 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.690047 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c963df6f-c03e-407e-8d8d-355141c73c40-kube-api-access-stv98" pod "c963df6f-c03e-407e-8d8d-355141c73c40" (UID: "c963df6f-c03e-407e-8d8d-355141c73c40"). InnerVolumeSpecName "kube-api-access-stv98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:24.741632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.741577 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:24.789633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.789595 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stv98\" (UniqueName: \"kubernetes.io/projected/c963df6f-c03e-407e-8d8d-355141c73c40-kube-api-access-stv98\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:24.789633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.789624 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963df6f-c03e-407e-8d8d-355141c73c40-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:24.980026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.979933 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c"] Nov 02 08:18:24.986704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:24.986677 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b62c067-8ddd9b947-wk89c"] Nov 02 08:18:25.575371 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:25.575341 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" Nov 02 08:18:25.575811 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:25.575378 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz" event={"ID":"c963df6f-c03e-407e-8d8d-355141c73c40","Type":"ContainerDied","Data":"23692eb2d97a9cfb4a76548372b464a30ac1db9aada9b95277a2c3deb868be5f"} Nov 02 08:18:25.575811 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:25.575432 6160 scope.go:122] "RemoveContainer" containerID="f5056837d28a2b892b10c0dedfe71fb6a37bd72fd607f3356d77ea7cf2352961" Nov 02 08:18:25.583843 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:25.583816 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611" path="/var/lib/kubelet/pods/ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611/volumes" Nov 02 08:18:25.763598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:25.763564 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz"] Nov 02 08:18:25.779382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:25.779298 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:25.809641 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:25.809606 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b62c067-d45b7f8cf-mfhwz"] Nov 02 08:18:25.994268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:25.994238 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:26.165466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:26.165432 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:26.337420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:26.337336 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:27.585154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:27.585111 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c963df6f-c03e-407e-8d8d-355141c73c40" path="/var/lib/kubelet/pods/c963df6f-c03e-407e-8d8d-355141c73c40/volumes" Nov 02 08:18:29.185232 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.185147 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" podUID="bad6b514-7e68-475f-a3e2-5326aae05f55" containerName="payload-processing" containerID="cri-o://b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d" gracePeriod=25 Nov 02 08:18:29.299782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.299741 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" podUID="b1466642-6fe1-40b6-8346-eb2e95a1e9e3" containerName="payload-pre-processing" containerID="cri-o://7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6" gracePeriod=25 Nov 02 08:18:29.440074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.440011 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:18:29.538118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.538091 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad6b514-7e68-475f-a3e2-5326aae05f55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad6b514-7e68-475f-a3e2-5326aae05f55-plugins-config-volume\") pod \"bad6b514-7e68-475f-a3e2-5326aae05f55\" (UID: \"bad6b514-7e68-475f-a3e2-5326aae05f55\") " Nov 02 08:18:29.538244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.538134 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad6b514-7e68-475f-a3e2-5326aae05f55-kube-api-access-7p8xr\" (UniqueName: \"kubernetes.io/projected/bad6b514-7e68-475f-a3e2-5326aae05f55-kube-api-access-7p8xr\") pod \"bad6b514-7e68-475f-a3e2-5326aae05f55\" (UID: \"bad6b514-7e68-475f-a3e2-5326aae05f55\") " Nov 02 08:18:29.538544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.538516 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad6b514-7e68-475f-a3e2-5326aae05f55-plugins-config-volume" pod "bad6b514-7e68-475f-a3e2-5326aae05f55" (UID: "bad6b514-7e68-475f-a3e2-5326aae05f55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:29.540337 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.540312 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad6b514-7e68-475f-a3e2-5326aae05f55-kube-api-access-7p8xr" pod "bad6b514-7e68-475f-a3e2-5326aae05f55" (UID: "bad6b514-7e68-475f-a3e2-5326aae05f55"). InnerVolumeSpecName "kube-api-access-7p8xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:29.546747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.546713 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:18:29.600925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.600894 6160 generic.go:356] "Generic (PLEG): container finished" podID="bad6b514-7e68-475f-a3e2-5326aae05f55" containerID="b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d" exitCode=0 Nov 02 08:18:29.601070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.600942 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" Nov 02 08:18:29.601070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.600973 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" event={"ID":"bad6b514-7e68-475f-a3e2-5326aae05f55","Type":"ContainerDied","Data":"b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d"} Nov 02 08:18:29.601070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.601042 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb" event={"ID":"bad6b514-7e68-475f-a3e2-5326aae05f55","Type":"ContainerDied","Data":"52d57f8ef8dfa9557773734b382593bed4da9649774afa04ace857266f62d954"} Nov 02 08:18:29.601070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.601064 6160 scope.go:122] "RemoveContainer" containerID="b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d" Nov 02 08:18:29.602544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.602522 6160 generic.go:356] "Generic (PLEG): container finished" podID="b1466642-6fe1-40b6-8346-eb2e95a1e9e3" containerID="7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6" exitCode=0 Nov 02 08:18:29.602677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.602570 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" Nov 02 08:18:29.602677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.602581 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" event={"ID":"b1466642-6fe1-40b6-8346-eb2e95a1e9e3","Type":"ContainerDied","Data":"7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6"} Nov 02 08:18:29.602677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.602615 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb" event={"ID":"b1466642-6fe1-40b6-8346-eb2e95a1e9e3","Type":"ContainerDied","Data":"c3a38e4b563a6b4c4a0bd9c9db8762efdf3932414154a60b033d80db8bb66db4"} Nov 02 08:18:29.610609 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.610592 6160 scope.go:122] "RemoveContainer" containerID="b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d" Nov 02 08:18:29.610912 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:29.610893 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d\": container with ID starting with b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d not found: ID does not exist" containerID="b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d" Nov 02 08:18:29.610974 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.610923 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d"} err="failed to get container status \"b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d\": rpc error: code = NotFound desc = could not find container \"b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d\": container with ID starting with b6904167b05762d30a62fe13997950d4d165e74efbb53ad592d948ba03ee7e8d not found: ID does not exist" Nov 02 08:18:29.610974 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.610939 6160 scope.go:122] "RemoveContainer" containerID="7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6" Nov 02 08:18:29.621411 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.621396 6160 scope.go:122] "RemoveContainer" containerID="7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6" Nov 02 08:18:29.621704 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:29.621681 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6\": container with ID starting with 7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6 not found: ID does not exist" containerID="7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6" Nov 02 08:18:29.621753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.621713 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6"} err="failed to get container status \"7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6\": rpc error: code = NotFound desc = could not find container \"7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6\": container with ID starting with 7b47f9e91ae2ea8de343e7cca5a496a3c7132e15b5bf1aa8d4d8ec4645f7c8d6 not found: ID does not exist" Nov 02 08:18:29.639509 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.639490 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-plugins-config-volume\") pod \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\" (UID: \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\") " Nov 02 08:18:29.639665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.639642 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-kube-api-access-2kmvn\" (UniqueName: \"kubernetes.io/projected/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-kube-api-access-2kmvn\") pod \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\" (UID: \"b1466642-6fe1-40b6-8346-eb2e95a1e9e3\") " Nov 02 08:18:29.639872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.639853 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-plugins-config-volume" pod "b1466642-6fe1-40b6-8346-eb2e95a1e9e3" (UID: "b1466642-6fe1-40b6-8346-eb2e95a1e9e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:29.640002 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.639988 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8xr\" (UniqueName: \"kubernetes.io/projected/bad6b514-7e68-475f-a3e2-5326aae05f55-kube-api-access-7p8xr\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:29.640059 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.640005 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:29.640059 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.640015 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad6b514-7e68-475f-a3e2-5326aae05f55-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:29.641709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.641685 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-kube-api-access-2kmvn" pod "b1466642-6fe1-40b6-8346-eb2e95a1e9e3" (UID: "b1466642-6fe1-40b6-8346-eb2e95a1e9e3"). InnerVolumeSpecName "kube-api-access-2kmvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:29.740686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.740603 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kmvn\" (UniqueName: \"kubernetes.io/projected/b1466642-6fe1-40b6-8346-eb2e95a1e9e3-kube-api-access-2kmvn\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:29.759366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.759333 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb"] Nov 02 08:18:29.761501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:29.761469 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-53cc8e62-74b7cc9b55-w4cnb"] Nov 02 08:18:30.015098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:30.015013 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb"] Nov 02 08:18:30.019710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:30.019682 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53cc8e62-8498c6ddc5-jhlwb"] Nov 02 08:18:30.646629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:30.646577 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:30.844088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:30.844046 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:31.002519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.002482 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:31.035853 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.035823 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx"] Nov 02 08:18:31.036263 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036250 6160 cpu_manager.go:414] "containerMap: removing container" podUID="b1466642-6fe1-40b6-8346-eb2e95a1e9e3" containerName="payload-pre-processing" Nov 02 08:18:31.036312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036265 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1466642-6fe1-40b6-8346-eb2e95a1e9e3" containerName="payload-pre-processing" Nov 02 08:18:31.036312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036274 6160 cpu_manager.go:414] "containerMap: removing container" podUID="bad6b514-7e68-475f-a3e2-5326aae05f55" containerName="payload-processing" Nov 02 08:18:31.036312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036279 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad6b514-7e68-475f-a3e2-5326aae05f55" containerName="payload-processing" Nov 02 08:18:31.036312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036292 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c963df6f-c03e-407e-8d8d-355141c73c40" containerName="payload-pre-processing" Nov 02 08:18:31.036312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036297 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c963df6f-c03e-407e-8d8d-355141c73c40" containerName="payload-pre-processing" Nov 02 08:18:31.036312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036305 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2f96b945-6b30-4cf3-82d4-4a8118488cd7" containerName="maas-api" Nov 02 08:18:31.036312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036309 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f96b945-6b30-4cf3-82d4-4a8118488cd7" containerName="maas-api" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036327 6160 cpu_manager.go:414] "containerMap: removing container" podUID="ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611" containerName="payload-processing" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036331 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611" containerName="payload-processing" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036338 6160 cpu_manager.go:414] "containerMap: removing container" podUID="b160f5aa-0a01-4a17-8ada-51528bbdcc40" containerName="payload-processing" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036342 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b160f5aa-0a01-4a17-8ada-51528bbdcc40" containerName="payload-processing" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036405 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1466642-6fe1-40b6-8346-eb2e95a1e9e3" containerName="payload-pre-processing" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036412 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad6b514-7e68-475f-a3e2-5326aae05f55" containerName="payload-processing" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036419 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c963df6f-c03e-407e-8d8d-355141c73c40" containerName="payload-pre-processing" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036424 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f96b945-6b30-4cf3-82d4-4a8118488cd7" containerName="maas-api" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036433 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae58a388-efd5-4c7d-a0ea-ab2cd7b1d611" containerName="payload-processing" Nov 02 08:18:31.036519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.036440 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="b160f5aa-0a01-4a17-8ada-51528bbdcc40" containerName="payload-processing" Nov 02 08:18:31.040791 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.040773 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:31.090590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.090559 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-10d2fb\"" Nov 02 08:18:31.090769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.090702 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-10d2fb-dockercfg-jb2gg\"" Nov 02 08:18:31.093660 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.093635 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx"] Nov 02 08:18:31.152236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.152169 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b4ba4-e975-4a6d-addc-37a40849ef7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx\" (UID: \"202b4ba4-e975-4a6d-addc-37a40849ef7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:31.152452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.152428 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzfv9\" (UniqueName: \"kubernetes.io/projected/202b4ba4-e975-4a6d-addc-37a40849ef7f-kube-api-access-rzfv9\") pod \"payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx\" (UID: \"202b4ba4-e975-4a6d-addc-37a40849ef7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:31.181803 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.181767 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8"] Nov 02 08:18:31.186209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.186184 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:31.240352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.240315 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8"] Nov 02 08:18:31.253260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.253197 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzfv9\" (UniqueName: \"kubernetes.io/projected/202b4ba4-e975-4a6d-addc-37a40849ef7f-kube-api-access-rzfv9\") pod \"payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx\" (UID: \"202b4ba4-e975-4a6d-addc-37a40849ef7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:31.253260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.253238 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ftv\" (UniqueName: \"kubernetes.io/projected/0434f554-d0b4-4dfb-9d87-efa7039b344c-kube-api-access-k5ftv\") pod \"payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8\" (UID: \"0434f554-d0b4-4dfb-9d87-efa7039b344c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:31.253420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.253266 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b4ba4-e975-4a6d-addc-37a40849ef7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx\" (UID: \"202b4ba4-e975-4a6d-addc-37a40849ef7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:31.253464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.253440 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0434f554-d0b4-4dfb-9d87-efa7039b344c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8\" (UID: \"0434f554-d0b4-4dfb-9d87-efa7039b344c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:31.254002 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.253984 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b4ba4-e975-4a6d-addc-37a40849ef7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx\" (UID: \"202b4ba4-e975-4a6d-addc-37a40849ef7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:31.306309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.306278 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzfv9\" (UniqueName: \"kubernetes.io/projected/202b4ba4-e975-4a6d-addc-37a40849ef7f-kube-api-access-rzfv9\") pod \"payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx\" (UID: \"202b4ba4-e975-4a6d-addc-37a40849ef7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:31.353523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.353494 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:31.354880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.354857 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ftv\" (UniqueName: \"kubernetes.io/projected/0434f554-d0b4-4dfb-9d87-efa7039b344c-kube-api-access-k5ftv\") pod \"payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8\" (UID: \"0434f554-d0b4-4dfb-9d87-efa7039b344c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:31.354959 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.354923 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0434f554-d0b4-4dfb-9d87-efa7039b344c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8\" (UID: \"0434f554-d0b4-4dfb-9d87-efa7039b344c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:31.355627 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.355609 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0434f554-d0b4-4dfb-9d87-efa7039b344c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8\" (UID: \"0434f554-d0b4-4dfb-9d87-efa7039b344c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:31.418907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.418875 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ftv\" (UniqueName: \"kubernetes.io/projected/0434f554-d0b4-4dfb-9d87-efa7039b344c-kube-api-access-k5ftv\") pod \"payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8\" (UID: \"0434f554-d0b4-4dfb-9d87-efa7039b344c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:31.452915 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.452884 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:31.498862 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.498825 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:31.585861 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.585824 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1466642-6fe1-40b6-8346-eb2e95a1e9e3" path="/var/lib/kubelet/pods/b1466642-6fe1-40b6-8346-eb2e95a1e9e3/volumes" Nov 02 08:18:31.586481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.586462 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad6b514-7e68-475f-a3e2-5326aae05f55" path="/var/lib/kubelet/pods/bad6b514-7e68-475f-a3e2-5326aae05f55/volumes" Nov 02 08:18:31.755684 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.755654 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx"] Nov 02 08:18:31.756406 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:18:31.756381 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202b4ba4_e975_4a6d_addc_37a40849ef7f.slice/crio-0d46cc48654b1d1afc9ac7ec284dd52cc7a3fd6d4c1597e42078a03653fa62e0 WatchSource:0}: Error finding container 0d46cc48654b1d1afc9ac7ec284dd52cc7a3fd6d4c1597e42078a03653fa62e0: Status 404 returned error can't find the container with id 0d46cc48654b1d1afc9ac7ec284dd52cc7a3fd6d4c1597e42078a03653fa62e0 Nov 02 08:18:31.869601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:31.869565 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8"] Nov 02 08:18:31.873597 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:18:31.873570 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0434f554_d0b4_4dfb_9d87_efa7039b344c.slice/crio-10fdf757277b0668798c76caf7cb836897107044c631d1564dbe18732e4b6b5d WatchSource:0}: Error finding container 10fdf757277b0668798c76caf7cb836897107044c631d1564dbe18732e4b6b5d: Status 404 returned error can't find the container with id 10fdf757277b0668798c76caf7cb836897107044c631d1564dbe18732e4b6b5d Nov 02 08:18:32.623367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:32.623319 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" event={"ID":"0434f554-d0b4-4dfb-9d87-efa7039b344c","Type":"ContainerStarted","Data":"21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344"} Nov 02 08:18:32.623547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:32.623392 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" event={"ID":"0434f554-d0b4-4dfb-9d87-efa7039b344c","Type":"ContainerStarted","Data":"10fdf757277b0668798c76caf7cb836897107044c631d1564dbe18732e4b6b5d"} Nov 02 08:18:32.623547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:32.623429 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:32.625163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:32.625124 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" event={"ID":"202b4ba4-e975-4a6d-addc-37a40849ef7f","Type":"ContainerStarted","Data":"6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571"} Nov 02 08:18:32.625295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:32.625197 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:32.625295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:32.625215 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" event={"ID":"202b4ba4-e975-4a6d-addc-37a40849ef7f","Type":"ContainerStarted","Data":"0d46cc48654b1d1afc9ac7ec284dd52cc7a3fd6d4c1597e42078a03653fa62e0"} Nov 02 08:18:32.778792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:32.778730 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" podStartSLOduration=1.778716264 podStartE2EDuration="1.778716264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:32.730789624 +0000 UTC m=+711.284293821" watchObservedRunningTime="2026-11-02 08:18:32.778716264 +0000 UTC m=+711.332220469" Nov 02 08:18:32.839721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:32.839552 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" podStartSLOduration=2.839534016 podStartE2EDuration="2.839534016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:32.838850991 +0000 UTC m=+711.392355208" watchObservedRunningTime="2026-11-02 08:18:32.839534016 +0000 UTC m=+711.393038215" Nov 02 08:18:35.475058 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:35.475018 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:36.704727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:36.704696 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:37.104329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:37.104237 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:37.417411 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:37.417369 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:38.208793 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:38.208749 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:38.362726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:38.362680 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:38.591443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:38.591343 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:38.755338 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:38.755306 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:38.871992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:38.871911 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm"] Nov 02 08:18:38.872139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:38.872118 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" containerID="cri-o://fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b" gracePeriod=30 Nov 02 08:18:38.921437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:38.921409 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:39.101098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:39.101060 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:39.516735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:39.516702 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:39.821544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:39.821451 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:39.998681 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:39.998645 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:40.520919 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:40.520879 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:18:40.557066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:40.557035 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:40.741364 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:40.741318 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:43.457674 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:43.457636 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:43.634236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:43.634201 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:18:43.634422 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:43.634269 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:18:43.642123 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:43.642093 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:43.798615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:43.798518 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:44.099155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.099120 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:44.139102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.139071 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p"] Nov 02 08:18:44.150249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.150225 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:18:44.285943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.285914 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3242831-c651-4f6b-953f-32726b062cc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3242831-c651-4f6b-953f-32726b062cc4-istiod-ca-cert\") pod \"c3242831-c651-4f6b-953f-32726b062cc4\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " Nov 02 08:18:44.286123 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.285973 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-istio-token\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-istio-token\") pod \"c3242831-c651-4f6b-953f-32726b062cc4\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " Nov 02 08:18:44.286209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286159 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-envoy\") pod \"c3242831-c651-4f6b-953f-32726b062cc4\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " Nov 02 08:18:44.286258 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286241 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-socket\") pod \"c3242831-c651-4f6b-953f-32726b062cc4\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " Nov 02 08:18:44.286298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286281 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-certs\") pod \"c3242831-c651-4f6b-953f-32726b062cc4\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " Nov 02 08:18:44.286347 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286331 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3242831-c651-4f6b-953f-32726b062cc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3242831-c651-4f6b-953f-32726b062cc4-istio-podinfo\") pod \"c3242831-c651-4f6b-953f-32726b062cc4\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " Nov 02 08:18:44.286395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286366 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-data\") pod \"c3242831-c651-4f6b-953f-32726b062cc4\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " Nov 02 08:18:44.286395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286370 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3242831-c651-4f6b-953f-32726b062cc4-istiod-ca-cert" pod "c3242831-c651-4f6b-953f-32726b062cc4" (UID: "c3242831-c651-4f6b-953f-32726b062cc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:44.286497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286415 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-kube-api-access-rb4xd\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-kube-api-access-rb4xd\") pod \"c3242831-c651-4f6b-953f-32726b062cc4\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " Nov 02 08:18:44.286497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286464 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-credential-socket\") pod \"c3242831-c651-4f6b-953f-32726b062cc4\" (UID: \"c3242831-c651-4f6b-953f-32726b062cc4\") " Nov 02 08:18:44.286611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286521 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-socket" pod "c3242831-c651-4f6b-953f-32726b062cc4" (UID: "c3242831-c651-4f6b-953f-32726b062cc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:44.286611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286578 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-certs" pod "c3242831-c651-4f6b-953f-32726b062cc4" (UID: "c3242831-c651-4f6b-953f-32726b062cc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:44.286844 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286824 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-data" pod "c3242831-c651-4f6b-953f-32726b062cc4" (UID: "c3242831-c651-4f6b-953f-32726b062cc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:44.286844 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286834 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-credential-socket" pod "c3242831-c651-4f6b-953f-32726b062cc4" (UID: "c3242831-c651-4f6b-953f-32726b062cc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:44.286997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286907 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:44.286997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286928 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:44.286997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.286942 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3242831-c651-4f6b-953f-32726b062cc4-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:44.288402 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.288373 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-envoy" pod "c3242831-c651-4f6b-953f-32726b062cc4" (UID: "c3242831-c651-4f6b-953f-32726b062cc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:18:44.288637 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.288617 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3242831-c651-4f6b-953f-32726b062cc4-istio-podinfo" pod "c3242831-c651-4f6b-953f-32726b062cc4" (UID: "c3242831-c651-4f6b-953f-32726b062cc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:18:44.288701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.288687 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-istio-token" pod "c3242831-c651-4f6b-953f-32726b062cc4" (UID: "c3242831-c651-4f6b-953f-32726b062cc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:44.288835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.288818 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-kube-api-access-rb4xd" pod "c3242831-c651-4f6b-953f-32726b062cc4" (UID: "c3242831-c651-4f6b-953f-32726b062cc4"). InnerVolumeSpecName "kube-api-access-rb4xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:44.388574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.388485 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:44.388574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.388518 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3242831-c651-4f6b-953f-32726b062cc4-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:44.388574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.388531 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:44.388574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.388543 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb4xd\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-kube-api-access-rb4xd\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:44.388574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.388557 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3242831-c651-4f6b-953f-32726b062cc4-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:44.388574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.388569 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3242831-c651-4f6b-953f-32726b062cc4-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:44.537802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.537769 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:44.698730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.698692 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:44.703121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.703091 6160 generic.go:356] "Generic (PLEG): container finished" podID="c3242831-c651-4f6b-953f-32726b062cc4" containerID="fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b" exitCode=0 Nov 02 08:18:44.703307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.703141 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" Nov 02 08:18:44.703307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.703203 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" event={"ID":"c3242831-c651-4f6b-953f-32726b062cc4","Type":"ContainerDied","Data":"fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b"} Nov 02 08:18:44.703307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.703269 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm" event={"ID":"c3242831-c651-4f6b-953f-32726b062cc4","Type":"ContainerDied","Data":"b00f61ad322ab486fe9046af82322e17fb048d96f4db64a935e54702a926fb25"} Nov 02 08:18:44.703307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.703298 6160 scope.go:122] "RemoveContainer" containerID="fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b" Nov 02 08:18:44.717533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.717516 6160 scope.go:122] "RemoveContainer" containerID="ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda" Nov 02 08:18:44.726426 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.726408 6160 scope.go:122] "RemoveContainer" containerID="fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b" Nov 02 08:18:44.726701 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:44.726682 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b\": container with ID starting with fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b not found: ID does not exist" containerID="fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b" Nov 02 08:18:44.726775 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.726715 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b"} err="failed to get container status \"fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b\": rpc error: code = NotFound desc = could not find container \"fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b\": container with ID starting with fb57d3df5e3a449227fa498af00ef07d3c7018baa566f63756cd4f9c458f931b not found: ID does not exist" Nov 02 08:18:44.726775 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.726742 6160 scope.go:122] "RemoveContainer" containerID="ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda" Nov 02 08:18:44.727043 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:44.727017 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda\": container with ID starting with ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda not found: ID does not exist" containerID="ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda" Nov 02 08:18:44.727081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.727050 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda"} err="failed to get container status \"ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda\": rpc error: code = NotFound desc = could not find container \"ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda\": container with ID starting with ccb651bee68be6ec9e54b47b935a5afefdfda3f408cf35f2c647ca099fec6fda not found: ID does not exist" Nov 02 08:18:44.856922 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.856887 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm"] Nov 02 08:18:44.861664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:44.861636 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-53cc8e62-gw-openshift-default-56f87b5c65-clzsm"] Nov 02 08:18:45.583588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:45.583550 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3242831-c651-4f6b-953f-32726b062cc4" path="/var/lib/kubelet/pods/c3242831-c651-4f6b-953f-32726b062cc4/volumes" Nov 02 08:18:47.725712 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:47.725678 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:48.159601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.159518 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:48.335299 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.335264 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:48.550525 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.550493 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6"] Nov 02 08:18:48.550931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.550918 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" Nov 02 08:18:48.550980 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.550932 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" Nov 02 08:18:48.550980 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.550943 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" Nov 02 08:18:48.550980 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.550948 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" Nov 02 08:18:48.551088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.551011 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" Nov 02 08:18:48.551088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.551020 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3242831-c651-4f6b-953f-32726b062cc4" containerName="istio-proxy" Nov 02 08:18:48.555562 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.555544 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:18:48.608660 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.608628 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6"] Nov 02 08:18:48.651681 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.651649 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-9d77b6\"" Nov 02 08:18:48.651907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.651761 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-9d77b6-dockercfg-tqkgp\"" Nov 02 08:18:48.698512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.698480 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2"] Nov 02 08:18:48.702102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.702085 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:18:48.732575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.732541 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a252ee0e-cb81-4412-9e74-6f046758501d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6\" (UID: \"a252ee0e-cb81-4412-9e74-6f046758501d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:18:48.732988 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.732581 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdch\" (UniqueName: \"kubernetes.io/projected/a252ee0e-cb81-4412-9e74-6f046758501d-kube-api-access-cgdch\") pod \"payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6\" (UID: \"a252ee0e-cb81-4412-9e74-6f046758501d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:18:48.757750 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.757623 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2"] Nov 02 08:18:48.833548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.833451 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqzn\" (UniqueName: \"kubernetes.io/projected/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-kube-api-access-qcqzn\") pod \"payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2\" (UID: \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:18:48.833717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.833586 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2\" (UID: \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:18:48.833717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.833632 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a252ee0e-cb81-4412-9e74-6f046758501d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6\" (UID: \"a252ee0e-cb81-4412-9e74-6f046758501d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:18:48.833717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.833656 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdch\" (UniqueName: \"kubernetes.io/projected/a252ee0e-cb81-4412-9e74-6f046758501d-kube-api-access-cgdch\") pod \"payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6\" (UID: \"a252ee0e-cb81-4412-9e74-6f046758501d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:18:48.834331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.834313 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a252ee0e-cb81-4412-9e74-6f046758501d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6\" (UID: \"a252ee0e-cb81-4412-9e74-6f046758501d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:18:48.887466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.887439 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdch\" (UniqueName: \"kubernetes.io/projected/a252ee0e-cb81-4412-9e74-6f046758501d-kube-api-access-cgdch\") pod \"payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6\" (UID: \"a252ee0e-cb81-4412-9e74-6f046758501d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:18:48.935112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.935078 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqzn\" (UniqueName: \"kubernetes.io/projected/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-kube-api-access-qcqzn\") pod \"payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2\" (UID: \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:18:48.935288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.935161 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2\" (UID: \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:18:48.935838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.935803 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2\" (UID: \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:18:48.990113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:48.990086 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqzn\" (UniqueName: \"kubernetes.io/projected/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-kube-api-access-qcqzn\") pod \"payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2\" (UID: \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:18:49.015864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.015829 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:18:49.141535 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.141434 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc"] Nov 02 08:18:49.145142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.145076 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" podUID="6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a" containerName="payload-processing" containerID="cri-o://8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f" gracePeriod=25 Nov 02 08:18:49.148316 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.148290 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.169069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.169040 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:18:49.198300 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.197684 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-10d2fb-serving-cert\"" Nov 02 08:18:49.199202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.199157 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc"] Nov 02 08:18:49.238885 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.238856 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-metrics-certs\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.239039 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.238917 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.239516 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.239481 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81bdb8e-3bce-4589-b74e-29be4446dd50-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.239632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.239613 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hln\" (UniqueName: \"kubernetes.io/projected/d81bdb8e-3bce-4589-b74e-29be4446dd50-kube-api-access-g5hln\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.341510 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.341154 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-metrics-certs\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.341510 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.341291 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.341510 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.341408 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81bdb8e-3bce-4589-b74e-29be4446dd50-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.341510 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.341511 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hln\" (UniqueName: \"kubernetes.io/projected/d81bdb8e-3bce-4589-b74e-29be4446dd50-kube-api-access-g5hln\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.341985 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:49.341913 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-serving-cert: secret "maas-api-e2e-worker-w2-10d2fb-serving-cert" not found Nov 02 08:18:49.342100 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:49.341992 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls podName:d81bdb8e-3bce-4589-b74e-29be4446dd50 nodeName:}" failed. No retries permitted until 2026-11-02 08:18:49.841968413 +0000 UTC m=+728.395472589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls") pod "maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" (UID: "d81bdb8e-3bce-4589-b74e-29be4446dd50") : secret "maas-api-e2e-worker-w2-10d2fb-serving-cert" not found Nov 02 08:18:49.342598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.342573 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81bdb8e-3bce-4589-b74e-29be4446dd50-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.344792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.344772 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-metrics-certs\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.374707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.374666 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:49.381827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.381798 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2"] Nov 02 08:18:49.383638 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:18:49.383613 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57eaa6d_c5c0_4126_bd0c_1c6f59c86c8b.slice/crio-8ff000c7f32cd6da4a7ee54a72eca96c84871cf73794ec849d7d854260a1900c WatchSource:0}: Error finding container 8ff000c7f32cd6da4a7ee54a72eca96c84871cf73794ec849d7d854260a1900c: Status 404 returned error can't find the container with id 8ff000c7f32cd6da4a7ee54a72eca96c84871cf73794ec849d7d854260a1900c Nov 02 08:18:49.391217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.391198 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:18:49.396398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.396345 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hln\" (UniqueName: \"kubernetes.io/projected/d81bdb8e-3bce-4589-b74e-29be4446dd50-kube-api-access-g5hln\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.535581 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.535538 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:49.537994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.537972 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6"] Nov 02 08:18:49.539590 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:18:49.539568 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda252ee0e_cb81_4412_9e74_6f046758501d.slice/crio-a2494ccc400c362971843674d40a1bb68dcbf6cf5de441df2deedca8524aba6b WatchSource:0}: Error finding container a2494ccc400c362971843674d40a1bb68dcbf6cf5de441df2deedca8524aba6b: Status 404 returned error can't find the container with id a2494ccc400c362971843674d40a1bb68dcbf6cf5de441df2deedca8524aba6b Nov 02 08:18:49.543574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.543555 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-plugins-config-volume\") pod \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\" (UID: \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\") " Nov 02 08:18:49.543659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.543591 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-kube-api-access-qgdkr\" (UniqueName: \"kubernetes.io/projected/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-kube-api-access-qgdkr\") pod \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\" (UID: \"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a\") " Nov 02 08:18:49.543950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.543930 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-plugins-config-volume" pod "6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a" (UID: "6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:49.544023 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.543998 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:49.545908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.545880 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-kube-api-access-qgdkr" pod "6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a" (UID: "6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a"). InnerVolumeSpecName "kube-api-access-qgdkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:49.645114 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.645082 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgdkr\" (UniqueName: \"kubernetes.io/projected/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a-kube-api-access-qgdkr\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:49.698496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.698465 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:49.733317 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.733282 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" event={"ID":"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b","Type":"ContainerStarted","Data":"59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080"} Nov 02 08:18:49.733805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.733347 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:18:49.733805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.733374 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" event={"ID":"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b","Type":"ContainerStarted","Data":"8ff000c7f32cd6da4a7ee54a72eca96c84871cf73794ec849d7d854260a1900c"} Nov 02 08:18:49.734819 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.734791 6160 generic.go:356] "Generic (PLEG): container finished" podID="6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a" containerID="8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f" exitCode=0 Nov 02 08:18:49.734977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.734828 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" Nov 02 08:18:49.734977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.734877 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" event={"ID":"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a","Type":"ContainerDied","Data":"8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f"} Nov 02 08:18:49.734977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.734922 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p" event={"ID":"6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a","Type":"ContainerDied","Data":"ccc1b76eaccbcbfdd2072d9258c91ce11e13a0eccdc972b0b6c3e467b642a8f3"} Nov 02 08:18:49.734977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.734951 6160 scope.go:122] "RemoveContainer" containerID="8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f" Nov 02 08:18:49.738109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.737282 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" event={"ID":"a252ee0e-cb81-4412-9e74-6f046758501d","Type":"ContainerStarted","Data":"4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff"} Nov 02 08:18:49.738109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.737330 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" event={"ID":"a252ee0e-cb81-4412-9e74-6f046758501d","Type":"ContainerStarted","Data":"a2494ccc400c362971843674d40a1bb68dcbf6cf5de441df2deedca8524aba6b"} Nov 02 08:18:49.738109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.737378 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:18:49.744869 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.744851 6160 scope.go:122] "RemoveContainer" containerID="8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f" Nov 02 08:18:49.745159 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:49.745137 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f\": container with ID starting with 8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f not found: ID does not exist" containerID="8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f" Nov 02 08:18:49.745255 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.745193 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f"} err="failed to get container status \"8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f\": rpc error: code = NotFound desc = could not find container \"8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f\": container with ID starting with 8cb860f8f71794b8a132068e806c03a3f57cef3459f563d13cbfc59f5a869a0f not found: ID does not exist" Nov 02 08:18:49.839817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.839751 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" podStartSLOduration=1.8397360329999999 podStartE2EDuration="1.839736033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:49.838766021 +0000 UTC m=+728.392270214" watchObservedRunningTime="2026-11-02 08:18:49.839736033 +0000 UTC m=+728.393240231" Nov 02 08:18:49.846473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.846442 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.849014 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.848988 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls\") pod \"maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:49.850120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.850094 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:49.943338 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.943307 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p"] Nov 02 08:18:49.947344 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:49.947314 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16457c0e-6f9bf4dfb9-6sr7p"] Nov 02 08:18:50.056040 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:50.055916 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" podStartSLOduration=2.055897897 podStartE2EDuration="2.055897897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:50.054776792 +0000 UTC m=+728.608281000" watchObservedRunningTime="2026-11-02 08:18:50.055897897 +0000 UTC m=+728.609402094" Nov 02 08:18:50.127297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:50.127218 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:50.280952 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:50.280924 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:50.506220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:50.506187 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:50.543025 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:50.543000 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc"] Nov 02 08:18:50.544717 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:18:50.544687 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81bdb8e_3bce_4589_b74e_29be4446dd50.slice/crio-ee122800fb30dc60d95253288445949eea163b1c1303667139c9925ec8f88df5 WatchSource:0}: Error finding container ee122800fb30dc60d95253288445949eea163b1c1303667139c9925ec8f88df5: Status 404 returned error can't find the container with id ee122800fb30dc60d95253288445949eea163b1c1303667139c9925ec8f88df5 Nov 02 08:18:50.727735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:50.727704 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:50.744914 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:50.744881 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" event={"ID":"d81bdb8e-3bce-4589-b74e-29be4446dd50","Type":"ContainerStarted","Data":"ee122800fb30dc60d95253288445949eea163b1c1303667139c9925ec8f88df5"} Nov 02 08:18:51.583075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:51.583041 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a" path="/var/lib/kubelet/pods/6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a/volumes" Nov 02 08:18:51.753344 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:51.753302 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" event={"ID":"d81bdb8e-3bce-4589-b74e-29be4446dd50","Type":"ContainerStarted","Data":"e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8"} Nov 02 08:18:51.753759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:51.753379 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:51.848255 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:51.825787 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms"] Nov 02 08:18:51.891733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:51.891683 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" podStartSLOduration=1.924857411 podStartE2EDuration="2.891668405s" totalImagesPullingTime="966.810994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:51.889229422 +0000 UTC m=+730.442733619" watchObservedRunningTime="2026-11-02 08:18:51.891668405 +0000 UTC m=+730.445172601" Nov 02 08:18:52.102643 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:52.102553 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:52.254233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:52.254164 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:53.702327 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:53.702291 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z"] Nov 02 08:18:53.702772 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:53.702501 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" podUID="3ff6956f-c65a-4c0a-91ab-30a185c84019" containerName="maas-api" containerID="cri-o://d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db" gracePeriod=30 Nov 02 08:18:53.831668 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:53.831638 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:53.952701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:53.952640 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:18:54.092132 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.092099 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff6956f-c65a-4c0a-91ab-30a185c84019-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff6956f-c65a-4c0a-91ab-30a185c84019-openshift-service-ca\") pod \"3ff6956f-c65a-4c0a-91ab-30a185c84019\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " Nov 02 08:18:54.092346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.092196 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls\") pod \"3ff6956f-c65a-4c0a-91ab-30a185c84019\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " Nov 02 08:18:54.092346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.092245 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-metrics-certs\") pod \"3ff6956f-c65a-4c0a-91ab-30a185c84019\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " Nov 02 08:18:54.092346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.092298 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff6956f-c65a-4c0a-91ab-30a185c84019-kube-api-access-vkp7b\" (UniqueName: \"kubernetes.io/projected/3ff6956f-c65a-4c0a-91ab-30a185c84019-kube-api-access-vkp7b\") pod \"3ff6956f-c65a-4c0a-91ab-30a185c84019\" (UID: \"3ff6956f-c65a-4c0a-91ab-30a185c84019\") " Nov 02 08:18:54.092580 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.092552 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff6956f-c65a-4c0a-91ab-30a185c84019-openshift-service-ca" pod "3ff6956f-c65a-4c0a-91ab-30a185c84019" (UID: "3ff6956f-c65a-4c0a-91ab-30a185c84019"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:54.094610 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.094577 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff6956f-c65a-4c0a-91ab-30a185c84019-kube-api-access-vkp7b" pod "3ff6956f-c65a-4c0a-91ab-30a185c84019" (UID: "3ff6956f-c65a-4c0a-91ab-30a185c84019"). InnerVolumeSpecName "kube-api-access-vkp7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:54.094709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.094603 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls" pod "3ff6956f-c65a-4c0a-91ab-30a185c84019" (UID: "3ff6956f-c65a-4c0a-91ab-30a185c84019"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:18:54.094709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.094630 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-metrics-certs" pod "3ff6956f-c65a-4c0a-91ab-30a185c84019" (UID: "3ff6956f-c65a-4c0a-91ab-30a185c84019"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:18:54.193676 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.193638 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff6956f-c65a-4c0a-91ab-30a185c84019-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:54.193676 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.193668 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:54.193676 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.193677 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff6956f-c65a-4c0a-91ab-30a185c84019-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:54.193909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.193687 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkp7b\" (UniqueName: \"kubernetes.io/projected/3ff6956f-c65a-4c0a-91ab-30a185c84019-kube-api-access-vkp7b\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:54.773183 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.773148 6160 generic.go:356] "Generic (PLEG): container finished" podID="3ff6956f-c65a-4c0a-91ab-30a185c84019" containerID="d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db" exitCode=0 Nov 02 08:18:54.773605 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.773223 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" Nov 02 08:18:54.773605 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.773242 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" event={"ID":"3ff6956f-c65a-4c0a-91ab-30a185c84019","Type":"ContainerDied","Data":"d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db"} Nov 02 08:18:54.773605 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.773302 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z" event={"ID":"3ff6956f-c65a-4c0a-91ab-30a185c84019","Type":"ContainerDied","Data":"53b72e76c38b2f67e0d275a2da81f5d276c1c1818e9d4acd13112b0a905c9818"} Nov 02 08:18:54.773605 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.773329 6160 scope.go:122] "RemoveContainer" containerID="d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db" Nov 02 08:18:54.783517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.783492 6160 scope.go:122] "RemoveContainer" containerID="d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db" Nov 02 08:18:54.783812 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:54.783792 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db\": container with ID starting with d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db not found: ID does not exist" containerID="d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db" Nov 02 08:18:54.783883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.783819 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db"} err="failed to get container status \"d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db\": rpc error: code = NotFound desc = could not find container \"d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db\": container with ID starting with d4251ecd3a58dc28a73a5f73fe7a39958151c2d99d6300fbfde944e97fc881db not found: ID does not exist" Nov 02 08:18:54.813842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.813808 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:54.940343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.940313 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z"] Nov 02 08:18:54.954504 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.954470 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e734c1c-789c6b9db7-b2q2z"] Nov 02 08:18:54.994906 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:54.994873 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:55.182026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:55.181991 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:55.583332 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:55.583250 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff6956f-c65a-4c0a-91ab-30a185c84019" path="/var/lib/kubelet/pods/3ff6956f-c65a-4c0a-91ab-30a185c84019/volumes" Nov 02 08:18:56.779618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:56.779578 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:56.848800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:56.848728 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" podUID="33260340-d2c8-4ce3-8483-455a27450dbb" containerName="payload-pre-processing" containerID="cri-o://f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520" gracePeriod=25 Nov 02 08:18:57.112925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.112893 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:18:57.116110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.116088 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33260340-d2c8-4ce3-8483-455a27450dbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33260340-d2c8-4ce3-8483-455a27450dbb-plugins-config-volume\") pod \"33260340-d2c8-4ce3-8483-455a27450dbb\" (UID: \"33260340-d2c8-4ce3-8483-455a27450dbb\") " Nov 02 08:18:57.116328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.116310 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33260340-d2c8-4ce3-8483-455a27450dbb-kube-api-access-kj2hq\" (UniqueName: \"kubernetes.io/projected/33260340-d2c8-4ce3-8483-455a27450dbb-kube-api-access-kj2hq\") pod \"33260340-d2c8-4ce3-8483-455a27450dbb\" (UID: \"33260340-d2c8-4ce3-8483-455a27450dbb\") " Nov 02 08:18:57.116418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.116393 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33260340-d2c8-4ce3-8483-455a27450dbb-plugins-config-volume" pod "33260340-d2c8-4ce3-8483-455a27450dbb" (UID: "33260340-d2c8-4ce3-8483-455a27450dbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:18:57.116634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.116616 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33260340-d2c8-4ce3-8483-455a27450dbb-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:57.118712 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.118686 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33260340-d2c8-4ce3-8483-455a27450dbb-kube-api-access-kj2hq" pod "33260340-d2c8-4ce3-8483-455a27450dbb" (UID: "33260340-d2c8-4ce3-8483-455a27450dbb"). InnerVolumeSpecName "kube-api-access-kj2hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:18:57.217434 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.217395 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj2hq\" (UniqueName: \"kubernetes.io/projected/33260340-d2c8-4ce3-8483-455a27450dbb-kube-api-access-kj2hq\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:18:57.504710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.504662 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:57.764395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.764316 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:18:57.779098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.779064 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:57.792812 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.792779 6160 generic.go:356] "Generic (PLEG): container finished" podID="33260340-d2c8-4ce3-8483-455a27450dbb" containerID="f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520" exitCode=0 Nov 02 08:18:57.793237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.792833 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" Nov 02 08:18:57.793237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.792872 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" event={"ID":"33260340-d2c8-4ce3-8483-455a27450dbb","Type":"ContainerDied","Data":"f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520"} Nov 02 08:18:57.793237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.792920 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms" event={"ID":"33260340-d2c8-4ce3-8483-455a27450dbb","Type":"ContainerDied","Data":"2ddf1eca8d38ba72137273f4d6726f59887db14a382990640dd4ed75126ef96b"} Nov 02 08:18:57.793237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.792943 6160 scope.go:122] "RemoveContainer" containerID="f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520" Nov 02 08:18:57.802337 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.802314 6160 scope.go:122] "RemoveContainer" containerID="f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520" Nov 02 08:18:57.802637 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:18:57.802610 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520\": container with ID starting with f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520 not found: ID does not exist" containerID="f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520" Nov 02 08:18:57.802709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.802647 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520"} err="failed to get container status \"f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520\": rpc error: code = NotFound desc = could not find container \"f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520\": container with ID starting with f60c6604ef3711b500d6e7f0713662549521ca5188852296148223cdee9d7520 not found: ID does not exist" Nov 02 08:18:57.976551 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.976510 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms"] Nov 02 08:18:57.991582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:57.991545 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e734c1c-57f65d68bc-wl6ms"] Nov 02 08:18:58.037044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:58.036951 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2"] Nov 02 08:18:58.037240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:58.037211 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" containerID="cri-o://e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545" gracePeriod=30 Nov 02 08:18:58.112425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:58.112391 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:58.848884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:58.848850 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:59.322882 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:59.322842 6160 patch_prober.go:29] interesting pod/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:18:59.323076 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:59.322910 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:18:59.510169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:59.510132 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:59.583454 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:59.583363 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33260340-d2c8-4ce3-8483-455a27450dbb" path="/var/lib/kubelet/pods/33260340-d2c8-4ce3-8483-455a27450dbb/volumes" Nov 02 08:18:59.723553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:18:59.723520 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:00.563848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.563811 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:00.749633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.749607 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:19:00.749802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.749668 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:19:00.768227 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.768158 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:00.956393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.956357 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86"] Nov 02 08:19:00.956867 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.956848 6160 cpu_manager.go:414] "containerMap: removing container" podUID="6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a" containerName="payload-processing" Nov 02 08:19:00.956963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.956869 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a" containerName="payload-processing" Nov 02 08:19:00.956963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.956891 6160 cpu_manager.go:414] "containerMap: removing container" podUID="33260340-d2c8-4ce3-8483-455a27450dbb" containerName="payload-pre-processing" Nov 02 08:19:00.956963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.956898 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33260340-d2c8-4ce3-8483-455a27450dbb" containerName="payload-pre-processing" Nov 02 08:19:00.956963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.956921 6160 cpu_manager.go:414] "containerMap: removing container" podUID="3ff6956f-c65a-4c0a-91ab-30a185c84019" containerName="maas-api" Nov 02 08:19:00.956963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.956929 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff6956f-c65a-4c0a-91ab-30a185c84019" containerName="maas-api" Nov 02 08:19:00.957231 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.957042 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bae3ef5-b8b8-4cc8-bb22-5796c7242d7a" containerName="payload-processing" Nov 02 08:19:00.957231 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.957057 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="33260340-d2c8-4ce3-8483-455a27450dbb" containerName="payload-pre-processing" Nov 02 08:19:00.957231 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.957069 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff6956f-c65a-4c0a-91ab-30a185c84019" containerName="maas-api" Nov 02 08:19:00.962230 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:00.962136 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:01.011961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.011912 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-108cf9\"" Nov 02 08:19:01.012434 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.012408 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-108cf9-dockercfg-798dv\"" Nov 02 08:19:01.014878 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.014850 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86"] Nov 02 08:19:01.056549 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.056513 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca996665-95bf-48ec-982c-73306576f87e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86\" (UID: \"ca996665-95bf-48ec-982c-73306576f87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:01.056716 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.056588 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbtd\" (UniqueName: \"kubernetes.io/projected/ca996665-95bf-48ec-982c-73306576f87e-kube-api-access-qfbtd\") pod \"payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86\" (UID: \"ca996665-95bf-48ec-982c-73306576f87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:01.101571 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.101539 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg"] Nov 02 08:19:01.105460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.105437 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:01.157280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.157242 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9887717-ef97-4158-95c7-dc9149c57caf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg\" (UID: \"f9887717-ef97-4158-95c7-dc9149c57caf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:01.157474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.157380 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca996665-95bf-48ec-982c-73306576f87e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86\" (UID: \"ca996665-95bf-48ec-982c-73306576f87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:01.157474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.157460 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwmlm\" (UniqueName: \"kubernetes.io/projected/f9887717-ef97-4158-95c7-dc9149c57caf-kube-api-access-zwmlm\") pod \"payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg\" (UID: \"f9887717-ef97-4158-95c7-dc9149c57caf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:01.157603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.157537 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbtd\" (UniqueName: \"kubernetes.io/projected/ca996665-95bf-48ec-982c-73306576f87e-kube-api-access-qfbtd\") pod \"payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86\" (UID: \"ca996665-95bf-48ec-982c-73306576f87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:01.158252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.158226 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca996665-95bf-48ec-982c-73306576f87e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86\" (UID: \"ca996665-95bf-48ec-982c-73306576f87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:01.159440 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.159416 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg"] Nov 02 08:19:01.211202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.211109 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbtd\" (UniqueName: \"kubernetes.io/projected/ca996665-95bf-48ec-982c-73306576f87e-kube-api-access-qfbtd\") pod \"payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86\" (UID: \"ca996665-95bf-48ec-982c-73306576f87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:01.258309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.258272 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwmlm\" (UniqueName: \"kubernetes.io/projected/f9887717-ef97-4158-95c7-dc9149c57caf-kube-api-access-zwmlm\") pod \"payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg\" (UID: \"f9887717-ef97-4158-95c7-dc9149c57caf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:01.258454 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.258362 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9887717-ef97-4158-95c7-dc9149c57caf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg\" (UID: \"f9887717-ef97-4158-95c7-dc9149c57caf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:01.259007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.258989 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9887717-ef97-4158-95c7-dc9149c57caf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg\" (UID: \"f9887717-ef97-4158-95c7-dc9149c57caf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:01.279664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.279636 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:01.312395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.312369 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwmlm\" (UniqueName: \"kubernetes.io/projected/f9887717-ef97-4158-95c7-dc9149c57caf-kube-api-access-zwmlm\") pod \"payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg\" (UID: \"f9887717-ef97-4158-95c7-dc9149c57caf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:01.420575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.420539 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:01.648911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.648883 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86"] Nov 02 08:19:01.651401 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:19:01.651375 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca996665_95bf_48ec_982c_73306576f87e.slice/crio-063655331f8192d9006b09c80577e29b2b58056e73f4adee73421ec9d5653ccb WatchSource:0}: Error finding container 063655331f8192d9006b09c80577e29b2b58056e73f4adee73421ec9d5653ccb: Status 404 returned error can't find the container with id 063655331f8192d9006b09c80577e29b2b58056e73f4adee73421ec9d5653ccb Nov 02 08:19:01.789820 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.789792 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg"] Nov 02 08:19:01.791770 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:19:01.791742 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9887717_ef97_4158_95c7_dc9149c57caf.slice/crio-051990674b6f2ea50b34ae272602cf471eff849cad112a5fc395c6041042496f WatchSource:0}: Error finding container 051990674b6f2ea50b34ae272602cf471eff849cad112a5fc395c6041042496f: Status 404 returned error can't find the container with id 051990674b6f2ea50b34ae272602cf471eff849cad112a5fc395c6041042496f Nov 02 08:19:01.818417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.818382 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" event={"ID":"ca996665-95bf-48ec-982c-73306576f87e","Type":"ContainerStarted","Data":"28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6"} Nov 02 08:19:01.818553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.818540 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:01.818629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.818572 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" event={"ID":"ca996665-95bf-48ec-982c-73306576f87e","Type":"ContainerStarted","Data":"063655331f8192d9006b09c80577e29b2b58056e73f4adee73421ec9d5653ccb"} Nov 02 08:19:01.819657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.819638 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" event={"ID":"f9887717-ef97-4158-95c7-dc9149c57caf","Type":"ContainerStarted","Data":"051990674b6f2ea50b34ae272602cf471eff849cad112a5fc395c6041042496f"} Nov 02 08:19:01.928142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.928086 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" podStartSLOduration=1.9280653399999998 podStartE2EDuration="1.92806534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:01.925961123 +0000 UTC m=+740.479465323" watchObservedRunningTime="2026-11-02 08:19:01.92806534 +0000 UTC m=+740.481569537" Nov 02 08:19:01.934982 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:01.934951 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:02.098613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:02.098525 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:02.826822 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:02.826783 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" event={"ID":"f9887717-ef97-4158-95c7-dc9149c57caf","Type":"ContainerStarted","Data":"390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271"} Nov 02 08:19:02.827277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:02.827018 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:02.934026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:02.933975 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" podStartSLOduration=1.933962304 podStartE2EDuration="1.933962304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:02.933739445 +0000 UTC m=+741.487243659" watchObservedRunningTime="2026-11-02 08:19:02.933962304 +0000 UTC m=+741.487466500" Nov 02 08:19:03.030814 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.030782 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:03.637975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.637944 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:03.709033 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.709008 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:19:03.783629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.783601 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-istio-token\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-istio-token\") pod \"50451fd9-2f4e-4871-ac49-b049890a1329\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " Nov 02 08:19:03.783794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.783648 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-credential-socket\") pod \"50451fd9-2f4e-4871-ac49-b049890a1329\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " Nov 02 08:19:03.783794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.783679 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50451fd9-2f4e-4871-ac49-b049890a1329-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50451fd9-2f4e-4871-ac49-b049890a1329-istiod-ca-cert\") pod \"50451fd9-2f4e-4871-ac49-b049890a1329\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " Nov 02 08:19:03.783794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.783707 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-data\") pod \"50451fd9-2f4e-4871-ac49-b049890a1329\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " Nov 02 08:19:03.783794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.783732 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-kube-api-access-jjm2j\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-kube-api-access-jjm2j\") pod \"50451fd9-2f4e-4871-ac49-b049890a1329\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " Nov 02 08:19:03.784017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.783815 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-envoy\") pod \"50451fd9-2f4e-4871-ac49-b049890a1329\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " Nov 02 08:19:03.784017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.783866 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50451fd9-2f4e-4871-ac49-b049890a1329-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50451fd9-2f4e-4871-ac49-b049890a1329-istio-podinfo\") pod \"50451fd9-2f4e-4871-ac49-b049890a1329\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " Nov 02 08:19:03.784017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.783901 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-certs\") pod \"50451fd9-2f4e-4871-ac49-b049890a1329\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " Nov 02 08:19:03.784017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.783944 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-socket\") pod \"50451fd9-2f4e-4871-ac49-b049890a1329\" (UID: \"50451fd9-2f4e-4871-ac49-b049890a1329\") " Nov 02 08:19:03.784252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.784043 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-credential-socket" pod "50451fd9-2f4e-4871-ac49-b049890a1329" (UID: "50451fd9-2f4e-4871-ac49-b049890a1329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:03.784252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.784057 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-data" pod "50451fd9-2f4e-4871-ac49-b049890a1329" (UID: "50451fd9-2f4e-4871-ac49-b049890a1329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:03.784252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.784120 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50451fd9-2f4e-4871-ac49-b049890a1329-istiod-ca-cert" pod "50451fd9-2f4e-4871-ac49-b049890a1329" (UID: "50451fd9-2f4e-4871-ac49-b049890a1329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:19:03.784413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.784289 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-certs" pod "50451fd9-2f4e-4871-ac49-b049890a1329" (UID: "50451fd9-2f4e-4871-ac49-b049890a1329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:03.784470 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.784432 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:03.784470 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.784439 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-socket" pod "50451fd9-2f4e-4871-ac49-b049890a1329" (UID: "50451fd9-2f4e-4871-ac49-b049890a1329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:03.784470 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.784451 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50451fd9-2f4e-4871-ac49-b049890a1329-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:03.784470 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.784464 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:03.784825 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.784477 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:03.786142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.786114 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-istio-token" pod "50451fd9-2f4e-4871-ac49-b049890a1329" (UID: "50451fd9-2f4e-4871-ac49-b049890a1329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:19:03.786277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.786234 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50451fd9-2f4e-4871-ac49-b049890a1329-istio-podinfo" pod "50451fd9-2f4e-4871-ac49-b049890a1329" (UID: "50451fd9-2f4e-4871-ac49-b049890a1329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:19:03.786353 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.786332 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-envoy" pod "50451fd9-2f4e-4871-ac49-b049890a1329" (UID: "50451fd9-2f4e-4871-ac49-b049890a1329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:03.786500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.786471 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-kube-api-access-jjm2j" pod "50451fd9-2f4e-4871-ac49-b049890a1329" (UID: "50451fd9-2f4e-4871-ac49-b049890a1329"). InnerVolumeSpecName "kube-api-access-jjm2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:19:03.806755 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.806680 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:03.833467 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.833436 6160 generic.go:356] "Generic (PLEG): container finished" podID="50451fd9-2f4e-4871-ac49-b049890a1329" containerID="e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545" exitCode=0 Nov 02 08:19:03.833816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.833489 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" Nov 02 08:19:03.833816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.833518 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" event={"ID":"50451fd9-2f4e-4871-ac49-b049890a1329","Type":"ContainerDied","Data":"e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545"} Nov 02 08:19:03.833816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.833567 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2" event={"ID":"50451fd9-2f4e-4871-ac49-b049890a1329","Type":"ContainerDied","Data":"8a00255bc5a625fd6ecbc4ecd57d426cda350237ef4650fa27cf27111683b87a"} Nov 02 08:19:03.833816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.833582 6160 scope.go:122] "RemoveContainer" containerID="e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545" Nov 02 08:19:03.843983 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.843960 6160 scope.go:122] "RemoveContainer" containerID="0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10" Nov 02 08:19:03.852657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.852622 6160 scope.go:122] "RemoveContainer" containerID="e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545" Nov 02 08:19:03.852881 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:19:03.852866 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545\": container with ID starting with e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545 not found: ID does not exist" containerID="e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545" Nov 02 08:19:03.852934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.852889 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545"} err="failed to get container status \"e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545\": rpc error: code = NotFound desc = could not find container \"e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545\": container with ID starting with e262b837cafeca589710aa4b7da3fbe78e9a925b4dcc7e439044cad565231545 not found: ID does not exist" Nov 02 08:19:03.852934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.852905 6160 scope.go:122] "RemoveContainer" containerID="0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10" Nov 02 08:19:03.853158 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:19:03.853141 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10\": container with ID starting with 0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10 not found: ID does not exist" containerID="0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10" Nov 02 08:19:03.853216 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.853165 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10"} err="failed to get container status \"0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10\": rpc error: code = NotFound desc = could not find container \"0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10\": container with ID starting with 0b3cab36c7cacb2a06157ec49e54fbf343fa61d9e912e27a395a291fe0971d10 not found: ID does not exist" Nov 02 08:19:03.884992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.884968 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:03.885119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.884994 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjm2j\" (UniqueName: \"kubernetes.io/projected/50451fd9-2f4e-4871-ac49-b049890a1329-kube-api-access-jjm2j\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:03.885119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.885012 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:03.885119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.885024 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50451fd9-2f4e-4871-ac49-b049890a1329-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:03.885119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.885038 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50451fd9-2f4e-4871-ac49-b049890a1329-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:19:03.977378 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.977326 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2"] Nov 02 08:19:03.979761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.979735 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-16457c0e-openshift-default-68984cb5df-gj9s2"] Nov 02 08:19:03.985364 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:03.985328 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:05.583910 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:05.583879 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" path="/var/lib/kubelet/pods/50451fd9-2f4e-4871-ac49-b049890a1329/volumes" Nov 02 08:19:06.774349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:06.774303 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:06.977733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:06.977658 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:07.068118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.068024 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7"] Nov 02 08:19:07.068596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.068575 6160 cpu_manager.go:414] "containerMap: removing container" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" Nov 02 08:19:07.068701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.068596 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" Nov 02 08:19:07.068759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.068728 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" Nov 02 08:19:07.068759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.068745 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" Nov 02 08:19:07.073550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.073531 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.114558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.114532 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.114669 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.114568 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08381029-9e19-447b-93cb-7722244cad27-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.114669 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.114595 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-metrics-certs\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.114770 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.114749 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngrrx\" (UniqueName: \"kubernetes.io/projected/08381029-9e19-447b-93cb-7722244cad27-kube-api-access-ngrrx\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.122158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.122138 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-9d77b6-serving-cert\"" Nov 02 08:19:07.137312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.137278 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:07.170970 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.170944 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7"] Nov 02 08:19:07.216135 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.216102 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.216351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.216159 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08381029-9e19-447b-93cb-7722244cad27-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.216351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.216226 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-metrics-certs\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.216351 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:19:07.216288 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-serving-cert: secret "maas-api-e2e-worker-w5-9d77b6-serving-cert" not found Nov 02 08:19:07.216351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.216339 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngrrx\" (UniqueName: \"kubernetes.io/projected/08381029-9e19-447b-93cb-7722244cad27-kube-api-access-ngrrx\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.216577 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:19:07.216364 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls podName:08381029-9e19-447b-93cb-7722244cad27 nodeName:}" failed. No retries permitted until 2026-11-02 08:19:07.716342914 +0000 UTC m=+746.269847093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls") pod "maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" (UID: "08381029-9e19-447b-93cb-7722244cad27") : secret "maas-api-e2e-worker-w5-9d77b6-serving-cert" not found Nov 02 08:19:07.217021 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.216991 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08381029-9e19-447b-93cb-7722244cad27-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.218953 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.218929 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-metrics-certs\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.269737 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.269702 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngrrx\" (UniqueName: \"kubernetes.io/projected/08381029-9e19-447b-93cb-7722244cad27-kube-api-access-ngrrx\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.286365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.286339 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk"] Nov 02 08:19:07.286815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.286803 6160 cpu_manager.go:414] "containerMap: removing container" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" Nov 02 08:19:07.286877 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.286816 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50451fd9-2f4e-4871-ac49-b049890a1329" containerName="istio-proxy" Nov 02 08:19:07.290050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.290035 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:07.307984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.307955 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:07.317240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.317213 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9rz2\" (UniqueName: \"kubernetes.io/projected/1b02178e-9684-47f3-b053-cc9b96353d6a-kube-api-access-z9rz2\") pod \"payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk\" (UID: \"1b02178e-9684-47f3-b053-cc9b96353d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:07.317350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.317258 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02178e-9684-47f3-b053-cc9b96353d6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk\" (UID: \"1b02178e-9684-47f3-b053-cc9b96353d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:07.341537 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.341377 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-096685\"" Nov 02 08:19:07.342022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.341988 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-096685-dockercfg-594f5\"" Nov 02 08:19:07.344283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.344260 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk"] Nov 02 08:19:07.418766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.418726 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9rz2\" (UniqueName: \"kubernetes.io/projected/1b02178e-9684-47f3-b053-cc9b96353d6a-kube-api-access-z9rz2\") pod \"payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk\" (UID: \"1b02178e-9684-47f3-b053-cc9b96353d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:07.418954 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.418785 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02178e-9684-47f3-b053-cc9b96353d6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk\" (UID: \"1b02178e-9684-47f3-b053-cc9b96353d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:07.419568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.419547 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02178e-9684-47f3-b053-cc9b96353d6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk\" (UID: \"1b02178e-9684-47f3-b053-cc9b96353d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:07.431782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.431750 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk"] Nov 02 08:19:07.435763 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.435748 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:07.469009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.468977 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:07.473262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.473236 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9rz2\" (UniqueName: \"kubernetes.io/projected/1b02178e-9684-47f3-b053-cc9b96353d6a-kube-api-access-z9rz2\") pod \"payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk\" (UID: \"1b02178e-9684-47f3-b053-cc9b96353d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:07.488979 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.488950 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk"] Nov 02 08:19:07.519965 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.519926 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvb89\" (UniqueName: \"kubernetes.io/projected/56e5538f-00d1-4b59-a85a-ec70ce18bf44-kube-api-access-cvb89\") pod \"payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk\" (UID: \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:07.520132 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.520049 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e5538f-00d1-4b59-a85a-ec70ce18bf44-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk\" (UID: \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:07.604158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.604076 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:07.621214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.621162 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvb89\" (UniqueName: \"kubernetes.io/projected/56e5538f-00d1-4b59-a85a-ec70ce18bf44-kube-api-access-cvb89\") pod \"payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk\" (UID: \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:07.621341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.621261 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e5538f-00d1-4b59-a85a-ec70ce18bf44-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk\" (UID: \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:07.622331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.622272 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e5538f-00d1-4b59-a85a-ec70ce18bf44-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk\" (UID: \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:07.631986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.631939 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:07.675591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.675561 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvb89\" (UniqueName: \"kubernetes.io/projected/56e5538f-00d1-4b59-a85a-ec70ce18bf44-kube-api-access-cvb89\") pod \"payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk\" (UID: \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:07.722392 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.722362 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.724812 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.724795 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:07.748483 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.748453 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:07.973202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.973156 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk"] Nov 02 08:19:07.974912 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:19:07.974877 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b02178e_9684_47f3_b053_cc9b96353d6a.slice/crio-1c61ccf138e02500493a4dd51902e6d9514c22ce55b183e9cae9a40845b1b67d WatchSource:0}: Error finding container 1c61ccf138e02500493a4dd51902e6d9514c22ce55b183e9cae9a40845b1b67d: Status 404 returned error can't find the container with id 1c61ccf138e02500493a4dd51902e6d9514c22ce55b183e9cae9a40845b1b67d Nov 02 08:19:07.986285 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:07.986256 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:08.115871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.115838 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk"] Nov 02 08:19:08.117134 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:19:08.117101 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e5538f_00d1_4b59_a85a_ec70ce18bf44.slice/crio-658e14a786f6706c1d386c343d2c5ae5f9495482c98574df33768d82b962c7ef WatchSource:0}: Error finding container 658e14a786f6706c1d386c343d2c5ae5f9495482c98574df33768d82b962c7ef: Status 404 returned error can't find the container with id 658e14a786f6706c1d386c343d2c5ae5f9495482c98574df33768d82b962c7ef Nov 02 08:19:08.357309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.357287 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7"] Nov 02 08:19:08.359092 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:19:08.359059 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08381029_9e19_447b_93cb_7722244cad27.slice/crio-573da68012da4a7b46b853f02c9759fc4676db696b95e0cbe907575b79096551 WatchSource:0}: Error finding container 573da68012da4a7b46b853f02c9759fc4676db696b95e0cbe907575b79096551: Status 404 returned error can't find the container with id 573da68012da4a7b46b853f02c9759fc4676db696b95e0cbe907575b79096551 Nov 02 08:19:08.865909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.865877 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" event={"ID":"1b02178e-9684-47f3-b053-cc9b96353d6a","Type":"ContainerStarted","Data":"3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e"} Nov 02 08:19:08.866133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.865932 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" event={"ID":"1b02178e-9684-47f3-b053-cc9b96353d6a","Type":"ContainerStarted","Data":"1c61ccf138e02500493a4dd51902e6d9514c22ce55b183e9cae9a40845b1b67d"} Nov 02 08:19:08.866133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.865983 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:08.867312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.867280 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" event={"ID":"08381029-9e19-447b-93cb-7722244cad27","Type":"ContainerStarted","Data":"573da68012da4a7b46b853f02c9759fc4676db696b95e0cbe907575b79096551"} Nov 02 08:19:08.868899 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.868878 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" event={"ID":"56e5538f-00d1-4b59-a85a-ec70ce18bf44","Type":"ContainerStarted","Data":"c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905"} Nov 02 08:19:08.868988 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.868910 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" event={"ID":"56e5538f-00d1-4b59-a85a-ec70ce18bf44","Type":"ContainerStarted","Data":"658e14a786f6706c1d386c343d2c5ae5f9495482c98574df33768d82b962c7ef"} Nov 02 08:19:08.869068 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.869015 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:08.973557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:08.973501 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" podStartSLOduration=1.973478791 podStartE2EDuration="1.973478791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:08.970644151 +0000 UTC m=+747.524148348" watchObservedRunningTime="2026-11-02 08:19:08.973478791 +0000 UTC m=+747.526982988" Nov 02 08:19:09.079137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:09.079085 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" podStartSLOduration=2.079069568 podStartE2EDuration="2.079069568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:09.075491301 +0000 UTC m=+747.628995498" watchObservedRunningTime="2026-11-02 08:19:09.079069568 +0000 UTC m=+747.632573765" Nov 02 08:19:09.876977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:09.876943 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" event={"ID":"08381029-9e19-447b-93cb-7722244cad27","Type":"ContainerStarted","Data":"f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d"} Nov 02 08:19:09.877367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:09.877349 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:09.986131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:09.986078 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" podStartSLOduration=2.158503237 podStartE2EDuration="2.986064754s" totalImagesPullingTime="827.561517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:09.983246772 +0000 UTC m=+748.536750968" watchObservedRunningTime="2026-11-02 08:19:09.986064754 +0000 UTC m=+748.539568950" Nov 02 08:19:10.520514 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:19:10.520478 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:19:12.628017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:12.627962 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:12.830696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:12.830666 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:19:13.836578 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:13.836549 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:19:14.213786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:14.213757 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:15.889931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:15.889904 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:19:17.665014 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:17.664975 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:19.879973 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:19.879945 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:19:19.880404 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:19.880019 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:19:22.208166 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.208126 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd"] Nov 02 08:19:22.213402 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.213377 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.263657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.263629 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-108cf9-serving-cert\"" Nov 02 08:19:22.312117 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.312085 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd"] Nov 02 08:19:22.373790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.373755 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fd97\" (UniqueName: \"kubernetes.io/projected/8a7f6ab0-5386-419a-b710-182ac04dd9c2-kube-api-access-6fd97\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.373948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.373874 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-metrics-certs\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.373948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.373910 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7f6ab0-5386-419a-b710-182ac04dd9c2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.374019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.374001 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.443220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.443170 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:22.474903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.474818 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fd97\" (UniqueName: \"kubernetes.io/projected/8a7f6ab0-5386-419a-b710-182ac04dd9c2-kube-api-access-6fd97\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.475075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.474920 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-metrics-certs\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.475075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.474952 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7f6ab0-5386-419a-b710-182ac04dd9c2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.475075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.474987 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.475759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.475689 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7f6ab0-5386-419a-b710-182ac04dd9c2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.477655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.477628 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-metrics-certs\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.477761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.477669 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.528488 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.528455 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fd97\" (UniqueName: \"kubernetes.io/projected/8a7f6ab0-5386-419a-b710-182ac04dd9c2-kube-api-access-6fd97\") pod \"maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.667078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.667014 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:22.826404 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.826313 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:22.829406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:22.829366 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:23.234731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:23.234701 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd"] Nov 02 08:19:23.236070 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:19:23.236047 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7f6ab0_5386_419a_b710_182ac04dd9c2.slice/crio-a3185c5cdbc48b825d70207fa0b9b87e23b4096d2cc56f0f575bd184bb52bc15 WatchSource:0}: Error finding container a3185c5cdbc48b825d70207fa0b9b87e23b4096d2cc56f0f575bd184bb52bc15: Status 404 returned error can't find the container with id a3185c5cdbc48b825d70207fa0b9b87e23b4096d2cc56f0f575bd184bb52bc15 Nov 02 08:19:23.958668 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:23.958635 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" event={"ID":"8a7f6ab0-5386-419a-b710-182ac04dd9c2","Type":"ContainerStarted","Data":"a3185c5cdbc48b825d70207fa0b9b87e23b4096d2cc56f0f575bd184bb52bc15"} Nov 02 08:19:24.966186 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:24.966138 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" event={"ID":"8a7f6ab0-5386-419a-b710-182ac04dd9c2","Type":"ContainerStarted","Data":"6054f1606f9bd6b42ad4ce2743569536039fa55217312229ae24f9a7d0a708d9"} Nov 02 08:19:24.966606 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:24.966214 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:25.073087 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:25.073029 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" podStartSLOduration=2.201488907 podStartE2EDuration="3.073009906s" totalImagesPullingTime="871.520999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:25.071939483 +0000 UTC m=+763.625443681" watchObservedRunningTime="2026-11-02 08:19:25.073009906 +0000 UTC m=+763.626514104" Nov 02 08:19:26.778911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:26.778875 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:28.843187 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:28.843140 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:29.135576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:29.135468 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:29.291043 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:29.291004 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:30.744210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:30.744154 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:30.979219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:30.979187 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:19:31.444413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:31.444383 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:32.255065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:32.255027 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:32.491439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:32.491390 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:34.564106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:34.564065 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:34.858885 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:34.858796 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:36.815387 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:36.815343 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:36.989979 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:36.989943 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:37.152139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:37.152057 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:37.324871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:37.324824 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:37.494475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:37.494440 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:39.551028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.550988 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt"] Nov 02 08:19:39.556218 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.556167 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.605861 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.605827 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-096685-serving-cert\"" Nov 02 08:19:39.639314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.639280 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.639314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.639315 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db39649-c218-4d8b-ac5d-2c76d5ef092e-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.639513 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.639467 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdf8x\" (UniqueName: \"kubernetes.io/projected/2db39649-c218-4d8b-ac5d-2c76d5ef092e-kube-api-access-zdf8x\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.639551 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.639538 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-metrics-certs\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.654556 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.654524 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt"] Nov 02 08:19:39.740953 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.740915 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.741118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.740960 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db39649-c218-4d8b-ac5d-2c76d5ef092e-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.741118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.741046 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdf8x\" (UniqueName: \"kubernetes.io/projected/2db39649-c218-4d8b-ac5d-2c76d5ef092e-kube-api-access-zdf8x\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.741118 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:19:39.741059 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-serving-cert: secret "maas-api-e2e-worker-w6-096685-serving-cert" not found Nov 02 08:19:39.741118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.741092 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-metrics-certs\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.741338 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:19:39.741121 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls podName:2db39649-c218-4d8b-ac5d-2c76d5ef092e nodeName:}" failed. No retries permitted until 2026-11-02 08:19:40.241099952 +0000 UTC m=+778.794604137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls") pod "maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" (UID: "2db39649-c218-4d8b-ac5d-2c76d5ef092e") : secret "maas-api-e2e-worker-w6-096685-serving-cert" not found Nov 02 08:19:39.741707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.741688 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db39649-c218-4d8b-ac5d-2c76d5ef092e-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.743738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.743715 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-metrics-certs\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.774188 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.774154 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:39.794684 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.794651 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdf8x\" (UniqueName: \"kubernetes.io/projected/2db39649-c218-4d8b-ac5d-2c76d5ef092e-kube-api-access-zdf8x\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:39.970049 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:39.970018 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:40.128992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.128954 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:40.246727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.246625 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:40.249209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.249160 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:40.284301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.284270 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:40.470032 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.469999 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:40.527340 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:19:40.527015 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:19:40.590916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.590878 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9"] Nov 02 08:19:40.596588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.596556 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:40.645761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.645729 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8e6407\"" Nov 02 08:19:40.645945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.645872 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8e6407-dockercfg-dz8f6\"" Nov 02 08:19:40.648942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.648911 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9"] Nov 02 08:19:40.650475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.650452 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f40e31-04e8-4217-9e47-c486d3a06d80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9\" (UID: \"e3f40e31-04e8-4217-9e47-c486d3a06d80\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:40.650595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.650574 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blzf8\" (UniqueName: \"kubernetes.io/projected/e3f40e31-04e8-4217-9e47-c486d3a06d80-kube-api-access-blzf8\") pod \"payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9\" (UID: \"e3f40e31-04e8-4217-9e47-c486d3a06d80\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:40.751460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.751423 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blzf8\" (UniqueName: \"kubernetes.io/projected/e3f40e31-04e8-4217-9e47-c486d3a06d80-kube-api-access-blzf8\") pod \"payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9\" (UID: \"e3f40e31-04e8-4217-9e47-c486d3a06d80\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:40.751623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.751553 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f40e31-04e8-4217-9e47-c486d3a06d80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9\" (UID: \"e3f40e31-04e8-4217-9e47-c486d3a06d80\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:40.752213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.752193 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f40e31-04e8-4217-9e47-c486d3a06d80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9\" (UID: \"e3f40e31-04e8-4217-9e47-c486d3a06d80\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:40.804950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.804880 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blzf8\" (UniqueName: \"kubernetes.io/projected/e3f40e31-04e8-4217-9e47-c486d3a06d80-kube-api-access-blzf8\") pod \"payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9\" (UID: \"e3f40e31-04e8-4217-9e47-c486d3a06d80\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:40.839968 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.839940 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt"] Nov 02 08:19:40.842145 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:19:40.842111 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db39649_c218_4d8b_ac5d_2c76d5ef092e.slice/crio-63786873be90eeedd46e7d0eb3de11c8c53289adca6a732e79e9431dfd11dd48 WatchSource:0}: Error finding container 63786873be90eeedd46e7d0eb3de11c8c53289adca6a732e79e9431dfd11dd48: Status 404 returned error can't find the container with id 63786873be90eeedd46e7d0eb3de11c8c53289adca6a732e79e9431dfd11dd48 Nov 02 08:19:40.912114 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:40.912080 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:41.066294 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:41.066202 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" event={"ID":"2db39649-c218-4d8b-ac5d-2c76d5ef092e","Type":"ContainerStarted","Data":"63786873be90eeedd46e7d0eb3de11c8c53289adca6a732e79e9431dfd11dd48"} Nov 02 08:19:41.280040 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:41.280012 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9"] Nov 02 08:19:41.282263 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:19:41.282233 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f40e31_04e8_4217_9e47_c486d3a06d80.slice/crio-44b4b6de75e8122a0e92cf86228072ebe9b14cce6568f2739a742c915a22ee0f WatchSource:0}: Error finding container 44b4b6de75e8122a0e92cf86228072ebe9b14cce6568f2739a742c915a22ee0f: Status 404 returned error can't find the container with id 44b4b6de75e8122a0e92cf86228072ebe9b14cce6568f2739a742c915a22ee0f Nov 02 08:19:42.073790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:42.073687 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" event={"ID":"2db39649-c218-4d8b-ac5d-2c76d5ef092e","Type":"ContainerStarted","Data":"4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005"} Nov 02 08:19:42.074254 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:42.073848 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:42.075490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:42.075464 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" event={"ID":"e3f40e31-04e8-4217-9e47-c486d3a06d80","Type":"ContainerStarted","Data":"47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1"} Nov 02 08:19:42.075613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:42.075505 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" event={"ID":"e3f40e31-04e8-4217-9e47-c486d3a06d80","Type":"ContainerStarted","Data":"44b4b6de75e8122a0e92cf86228072ebe9b14cce6568f2739a742c915a22ee0f"} Nov 02 08:19:42.075613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:42.075529 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:42.182385 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:42.182321 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" podStartSLOduration=2.341656591 podStartE2EDuration="3.182307173s" totalImagesPullingTime="840.650582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:42.179149187 +0000 UTC m=+780.732653383" watchObservedRunningTime="2026-11-02 08:19:42.182307173 +0000 UTC m=+780.735811369" Nov 02 08:19:42.296622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:42.296566 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" podStartSLOduration=2.29655258 podStartE2EDuration="2.29655258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:42.293967216 +0000 UTC m=+780.847471422" watchObservedRunningTime="2026-11-02 08:19:42.29655258 +0000 UTC m=+780.850056777" Nov 02 08:19:45.180093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:45.180051 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:48.087898 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:48.087869 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:19:50.336544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:50.336503 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:53.084748 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:53.084715 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:19:54.031154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:54.031116 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:54.744434 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:54.744400 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:54.906858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:54.906819 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:55.616521 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:55.616461 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:58.089233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:58.089170 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:58.237011 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:19:58.236975 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:00.729407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:00.729369 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:02.271105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:02.271062 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:02.552841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:02.552744 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:02.970219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:02.970156 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:05.957109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:05.957070 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:06.849902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:06.849857 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:10.251890 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:10.251849 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:10.520300 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:20:10.520209 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:20:11.153486 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:11.153447 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:14.367572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:14.367516 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:15.659545 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.659515 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc"] Nov 02 08:20:15.664613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.664596 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.762474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.762436 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8e6407-serving-cert\"" Nov 02 08:20:15.765093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.765070 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc"] Nov 02 08:20:15.800388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.800356 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.800540 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.800409 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-metrics-certs\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.800583 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.800527 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhd44\" (UniqueName: \"kubernetes.io/projected/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-kube-api-access-hhd44\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.800733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.800708 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.870940 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.870905 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:15.901818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.901781 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-metrics-certs\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.901979 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.901869 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhd44\" (UniqueName: \"kubernetes.io/projected/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-kube-api-access-hhd44\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.901979 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.901950 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.902093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.902028 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.902093 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:20:15.902048 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-serving-cert: secret "maas-api-e2e-worker-w4-8e6407-serving-cert" not found Nov 02 08:20:15.902226 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:20:15.902138 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls podName:f530dfdb-de1d-4ea4-b590-f4f953a96cc7 nodeName:}" failed. No retries permitted until 2026-11-02 08:20:16.402115224 +0000 UTC m=+814.955619419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls") pod "maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" (UID: "f530dfdb-de1d-4ea4-b590-f4f953a96cc7") : secret "maas-api-e2e-worker-w4-8e6407-serving-cert" not found Nov 02 08:20:15.902660 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.902634 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.904547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.904526 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-metrics-certs\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:15.958111 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:15.958080 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhd44\" (UniqueName: \"kubernetes.io/projected/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-kube-api-access-hhd44\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:16.045915 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:16.045883 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:16.195891 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:16.195846 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:16.407061 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:16.406969 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:16.409460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:16.409432 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:16.603083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:16.603048 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:16.973342 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:16.973318 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc"] Nov 02 08:20:16.975100 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:20:16.975070 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf530dfdb_de1d_4ea4_b590_f4f953a96cc7.slice/crio-fff0e4dd08357fa97099404e656777d099005c4db993b53c6d3f864b040df7bb WatchSource:0}: Error finding container fff0e4dd08357fa97099404e656777d099005c4db993b53c6d3f864b040df7bb: Status 404 returned error can't find the container with id fff0e4dd08357fa97099404e656777d099005c4db993b53c6d3f864b040df7bb Nov 02 08:20:17.283303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:17.283218 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" event={"ID":"f530dfdb-de1d-4ea4-b590-f4f953a96cc7","Type":"ContainerStarted","Data":"fff0e4dd08357fa97099404e656777d099005c4db993b53c6d3f864b040df7bb"} Nov 02 08:20:18.290946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:18.290911 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" event={"ID":"f530dfdb-de1d-4ea4-b590-f4f953a96cc7","Type":"ContainerStarted","Data":"9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50"} Nov 02 08:20:18.291453 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:18.291064 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:18.398761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:18.398703 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" podStartSLOduration=2.534763897 podStartE2EDuration="3.398687798s" totalImagesPullingTime="863.923901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:20:18.398365501 +0000 UTC m=+816.951869721" watchObservedRunningTime="2026-11-02 08:20:18.398687798 +0000 UTC m=+816.952191991" Nov 02 08:20:20.552596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:20.552511 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:20.973993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:20.973957 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:24.302737 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:24.302701 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:20:25.854829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:25.854790 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:29.565648 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:29.565607 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:30.237859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:30.237816 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:30.459654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:30.459620 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:31.073642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:31.073601 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:31.249286 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:31.249234 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:35.937378 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:35.937341 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:40.522168 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:20:40.522129 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:20:40.581865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:40.581829 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:41.621875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:41.621831 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:45.541728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:45.541690 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:50.272092 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:50.272055 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:54.734436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:54.734397 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:54.886543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:54.886508 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:55.129568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:55.129460 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:59.848923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:20:59.848884 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:02.628832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:02.628792 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:03.266925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:03.266848 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:03.841368 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:03.841327 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:04.013597 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:04.013554 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:05.139371 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:05.139330 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:06.847281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:06.847245 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:07.431487 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:07.431451 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:09.935463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:09.935427 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:10.520584 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:21:10.520542 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:21:11.898367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:11.898330 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:13.993810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:13.993773 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:14.734756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:14.734722 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:17.187881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:17.187845 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:17.612618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:17.612504 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:19.543608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:19.543565 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:22.779951 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:22.779914 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd"] Nov 02 08:21:22.780446 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:22.780133 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" containerID="cri-o://a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450" gracePeriod=30 Nov 02 08:21:24.312139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:24.312096 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:27.177227 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.177189 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9"] Nov 02 08:21:27.181954 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.181926 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.245810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.245775 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.245996 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.245832 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.245996 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.245899 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.245996 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.245942 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.246204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.246004 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.246204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.246113 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dz2g\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-kube-api-access-2dz2g\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.246204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.246142 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.246374 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.246212 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.246374 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.246235 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.278969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.278935 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d0628e-openshift-default-dockercfg-qvj7g\"" Nov 02 08:21:27.281818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.281790 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9"] Nov 02 08:21:27.347596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.347565 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.347596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.347600 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.347815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.347626 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.347815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.347665 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.347815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.347703 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.347815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.347754 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.347815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.347793 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.348070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.347887 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dz2g\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-kube-api-access-2dz2g\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.348070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.347933 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.348070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.348006 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.348388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.348368 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.348482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.348414 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.348662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.348639 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.348744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.348658 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.350248 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.350224 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.351016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.350991 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.401823 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.401793 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.402007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.401958 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dz2g\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-kube-api-access-2dz2g\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.497424 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.497337 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:27.921748 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.921703 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9"] Nov 02 08:21:27.956210 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:21:27.956144 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7a06ed_0628_42db_9682_2e3ec8e5bca7.slice/crio-4d8ee6adeae251317c80e89711b6a26c89df13c45633ba1eb97ba2db5570eb51 WatchSource:0}: Error finding container 4d8ee6adeae251317c80e89711b6a26c89df13c45633ba1eb97ba2db5570eb51: Status 404 returned error can't find the container with id 4d8ee6adeae251317c80e89711b6a26c89df13c45633ba1eb97ba2db5570eb51 Nov 02 08:21:27.960233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.960167 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:21:27.960335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.960278 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:21:27.960335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:27.960319 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:21:28.030900 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.030876 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:21:28.055156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055073 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-token\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-token\") pod \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " Nov 02 08:21:28.055156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055131 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-podinfo\") pod \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " Nov 02 08:21:28.055389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055197 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-envoy\") pod \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " Nov 02 08:21:28.055389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055226 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-certs\") pod \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " Nov 02 08:21:28.055389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055273 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-credential-socket\") pod \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " Nov 02 08:21:28.055389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055332 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-socket\") pod \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " Nov 02 08:21:28.055596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055394 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-kube-api-access-5ptf4\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-kube-api-access-5ptf4\") pod \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " Nov 02 08:21:28.055596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055432 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-data\") pod \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " Nov 02 08:21:28.055596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055477 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istiod-ca-cert\") pod \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\" (UID: \"23c2349b-4ccf-4e8b-8507-1b630e65fa30\") " Nov 02 08:21:28.055736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055689 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-credential-socket" pod "23c2349b-4ccf-4e8b-8507-1b630e65fa30" (UID: "23c2349b-4ccf-4e8b-8507-1b630e65fa30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:28.055961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055934 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:28.056041 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.055970 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istiod-ca-cert" pod "23c2349b-4ccf-4e8b-8507-1b630e65fa30" (UID: "23c2349b-4ccf-4e8b-8507-1b630e65fa30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:21:28.056228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.056155 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-certs" pod "23c2349b-4ccf-4e8b-8507-1b630e65fa30" (UID: "23c2349b-4ccf-4e8b-8507-1b630e65fa30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:28.056302 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.056252 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-socket" pod "23c2349b-4ccf-4e8b-8507-1b630e65fa30" (UID: "23c2349b-4ccf-4e8b-8507-1b630e65fa30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:28.056462 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.056442 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-data" pod "23c2349b-4ccf-4e8b-8507-1b630e65fa30" (UID: "23c2349b-4ccf-4e8b-8507-1b630e65fa30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:28.058083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.058054 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-envoy" pod "23c2349b-4ccf-4e8b-8507-1b630e65fa30" (UID: "23c2349b-4ccf-4e8b-8507-1b630e65fa30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:28.058196 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.058071 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-token" pod "23c2349b-4ccf-4e8b-8507-1b630e65fa30" (UID: "23c2349b-4ccf-4e8b-8507-1b630e65fa30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:21:28.059242 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.059205 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-kube-api-access-5ptf4" pod "23c2349b-4ccf-4e8b-8507-1b630e65fa30" (UID: "23c2349b-4ccf-4e8b-8507-1b630e65fa30"). InnerVolumeSpecName "kube-api-access-5ptf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:21:28.059338 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.059195 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-podinfo" pod "23c2349b-4ccf-4e8b-8507-1b630e65fa30" (UID: "23c2349b-4ccf-4e8b-8507-1b630e65fa30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:21:28.157115 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.157080 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:28.157115 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.157116 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:28.157381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.157133 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:28.157381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.157147 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ptf4\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-kube-api-access-5ptf4\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:28.157381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.157162 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:28.157381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.157220 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:28.157381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.157234 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:28.157381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.157248 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c2349b-4ccf-4e8b-8507-1b630e65fa30-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:21:28.732203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.724628 6160 generic.go:356] "Generic (PLEG): container finished" podID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerID="a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450" exitCode=0 Nov 02 08:21:28.732203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.724881 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" event={"ID":"23c2349b-4ccf-4e8b-8507-1b630e65fa30","Type":"ContainerDied","Data":"a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450"} Nov 02 08:21:28.732203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.724973 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" event={"ID":"23c2349b-4ccf-4e8b-8507-1b630e65fa30","Type":"ContainerDied","Data":"d812d59808eaef8bef8bdb5cecd119ca994e59f9a5131e9a934f3fec12d20ea6"} Nov 02 08:21:28.732203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.725011 6160 scope.go:122] "RemoveContainer" containerID="a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450" Nov 02 08:21:28.732203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.725339 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd" Nov 02 08:21:28.737886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.737858 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" event={"ID":"2c7a06ed-0628-42db-9682-2e3ec8e5bca7","Type":"ContainerStarted","Data":"edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f"} Nov 02 08:21:28.738035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.737907 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" event={"ID":"2c7a06ed-0628-42db-9682-2e3ec8e5bca7","Type":"ContainerStarted","Data":"4d8ee6adeae251317c80e89711b6a26c89df13c45633ba1eb97ba2db5570eb51"} Nov 02 08:21:28.746408 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.746382 6160 scope.go:122] "RemoveContainer" containerID="a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450" Nov 02 08:21:28.746693 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:21:28.746662 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450\": container with ID starting with a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450 not found: ID does not exist" containerID="a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450" Nov 02 08:21:28.746781 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.746704 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450"} err="failed to get container status \"a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450\": rpc error: code = NotFound desc = could not find container \"a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450\": container with ID starting with a1321c2b4f0759479acc769442400c69824ddae746bdf9dc525e4f1044449450 not found: ID does not exist" Nov 02 08:21:28.885989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.885946 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd"] Nov 02 08:21:28.935191 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.935140 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-skrdd"] Nov 02 08:21:28.995816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:28.995691 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" podStartSLOduration=1.9956703949999999 podStartE2EDuration="1.995670395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:28.994779868 +0000 UTC m=+887.548284086" watchObservedRunningTime="2026-11-02 08:21:28.995670395 +0000 UTC m=+887.549174594" Nov 02 08:21:29.032913 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:29.032874 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:29.497511 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:29.497475 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:29.498888 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:29.498861 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 08:21:29.499022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:29.498924 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 08:21:29.584229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:29.584192 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" path="/var/lib/kubelet/pods/23c2349b-4ccf-4e8b-8507-1b630e65fa30/volumes" Nov 02 08:21:29.997505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:29.997457 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:30.498218 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:30.498155 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 08:21:30.498399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:30.498247 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 08:21:31.234117 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.234081 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj"] Nov 02 08:21:31.234646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.234590 6160 cpu_manager.go:414] "containerMap: removing container" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" Nov 02 08:21:31.234646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.234603 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" Nov 02 08:21:31.234765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.234696 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c2349b-4ccf-4e8b-8507-1b630e65fa30" containerName="istio-proxy" Nov 02 08:21:31.240500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.240478 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.288373 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.288336 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwj79\" (UniqueName: \"kubernetes.io/projected/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-kube-api-access-cwj79\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.288373 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.288375 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-home\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.288648 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.288401 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-dshm\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.288648 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.288490 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tmp-dir\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.288721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.288678 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tls-certs\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.288761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.288721 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-model-cache\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.292387 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.292363 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj"] Nov 02 08:21:31.335778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.335712 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e6407\"/\"kube-root-ca.crt\"" Nov 02 08:21:31.336073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.335881 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e6407\"/\"default-dockercfg-zz4bc\"" Nov 02 08:21:31.336073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.335977 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e6407\"/\"simulator-w4-8e6407-kserve-self-signed-certs\"" Nov 02 08:21:31.336073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.336024 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e6407\"/\"openshift-service-ca.crt\"" Nov 02 08:21:31.354463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.354419 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:31.389841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.389806 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tls-certs\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.389841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.389848 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-model-cache\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.390058 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.389894 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwj79\" (UniqueName: \"kubernetes.io/projected/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-kube-api-access-cwj79\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.390058 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.389919 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-home\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.390058 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.389943 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-dshm\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.390058 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.389984 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tmp-dir\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.390530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.390474 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tmp-dir\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.390660 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.390587 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-model-cache\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.390728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.390662 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-home\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.392815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.392791 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-dshm\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.393733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.393713 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tls-certs\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.447621 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.447581 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwj79\" (UniqueName: \"kubernetes.io/projected/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-kube-api-access-cwj79\") pod \"simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.498226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.498130 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 08:21:31.498226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.498203 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 08:21:31.552087 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.552053 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:31.949200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:31.949156 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj"] Nov 02 08:21:31.951191 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:21:31.951130 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c8b3a4d_3ddb_46a7_9c8d_30e3d4f39db2.slice/crio-314d2e5c1a13e5607aa732391e7b655e3805971183bd7784b907e8b281ac9549 WatchSource:0}: Error finding container 314d2e5c1a13e5607aa732391e7b655e3805971183bd7784b907e8b281ac9549: Status 404 returned error can't find the container with id 314d2e5c1a13e5607aa732391e7b655e3805971183bd7784b907e8b281ac9549 Nov 02 08:21:32.498439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:32.498403 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 08:21:32.498860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:32.498457 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 08:21:32.766656 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:32.766562 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" event={"ID":"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2","Type":"ContainerStarted","Data":"b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09"} Nov 02 08:21:32.766656 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:32.766610 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" event={"ID":"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2","Type":"ContainerStarted","Data":"314d2e5c1a13e5607aa732391e7b655e3805971183bd7784b907e8b281ac9549"} Nov 02 08:21:32.884065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:32.884010 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" podStartSLOduration=1.883994556 podStartE2EDuration="1.883994556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:32.880956984 +0000 UTC m=+891.434461179" watchObservedRunningTime="2026-11-02 08:21:32.883994556 +0000 UTC m=+891.437498812" Nov 02 08:21:33.498154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:33.498116 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 08:21:33.498365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:33.498202 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 08:21:33.552919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:33.552886 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:33.842304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:33.842220 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:34.498026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:34.497991 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 08:21:34.498229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:34.498040 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 08:21:35.502358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:35.502323 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:35.502816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:35.502630 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:35.503413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:35.503394 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:21:38.558366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:38.558325 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:40.520760 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:21:40.520724 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:21:41.553239 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:41.553201 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:41.566497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:41.566468 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:41.702761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:41.702728 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:21:41.703284 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:41.703265 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:21:41.834617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:41.834534 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:21:43.230655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:43.230621 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:46.928207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:46.928152 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck"] Nov 02 08:21:46.933796 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:46.933771 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:46.982743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:46.982702 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e6ca80-openshift-default-dockercfg-kspcn\"" Nov 02 08:21:47.031873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.031841 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck"] Nov 02 08:21:47.070997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.070968 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prc7f\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-kube-api-access-prc7f\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.070997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.071002 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-credential-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.071233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.071079 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-certs\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.071233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.071137 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-data\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.071233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.071218 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-token\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.071339 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.071277 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-envoy\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.071339 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.071303 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d822da84-b8e1-43c1-b841-6a220c658d8c-istiod-ca-cert\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.071339 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.071323 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.071458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.071360 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-podinfo\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.171977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.171938 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-certs\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.171993 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-data\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172031 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-token\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172070 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-envoy\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172226 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d822da84-b8e1-43c1-b841-6a220c658d8c-istiod-ca-cert\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172279 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172305 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-podinfo\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172520 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172376 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prc7f\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-kube-api-access-prc7f\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172520 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172414 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-credential-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172520 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172447 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-data\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172413 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-certs\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172623 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172816 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-credential-socket\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.172906 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.172886 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d822da84-b8e1-43c1-b841-6a220c658d8c-istiod-ca-cert\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.174456 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.174436 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-envoy\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.174831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.174811 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-podinfo\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.225508 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.225421 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-token\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.225760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.225739 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prc7f\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-kube-api-access-prc7f\") pod \"e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.246546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.246524 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:47.674199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.674076 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck"] Nov 02 08:21:47.678734 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:21:47.678702 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd822da84_b8e1_43c1_b841_6a220c658d8c.slice/crio-4cfc4d4f247c1a7f8f369c2396244bbd591a7f001322698f44aea5f979f50bc2 WatchSource:0}: Error finding container 4cfc4d4f247c1a7f8f369c2396244bbd591a7f001322698f44aea5f979f50bc2: Status 404 returned error can't find the container with id 4cfc4d4f247c1a7f8f369c2396244bbd591a7f001322698f44aea5f979f50bc2 Nov 02 08:21:47.680690 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.680654 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:21:47.680787 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.680724 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:21:47.680787 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.680770 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:21:47.861949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.861909 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" event={"ID":"d822da84-b8e1-43c1-b841-6a220c658d8c","Type":"ContainerStarted","Data":"f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c"} Nov 02 08:21:47.862116 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.861971 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" event={"ID":"d822da84-b8e1-43c1-b841-6a220c658d8c","Type":"ContainerStarted","Data":"4cfc4d4f247c1a7f8f369c2396244bbd591a7f001322698f44aea5f979f50bc2"} Nov 02 08:21:47.976594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.976479 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" podStartSLOduration=1.9764629569999999 podStartE2EDuration="1.976462957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:47.975082028 +0000 UTC m=+906.528586226" watchObservedRunningTime="2026-11-02 08:21:47.976462957 +0000 UTC m=+906.529967153" Nov 02 08:21:47.999662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:47.999615 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:48.247009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:48.246919 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:48.248705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:48.248676 6160 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 08:21:48.248842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:48.248725 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 08:21:49.247738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:49.247698 6160 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 08:21:49.248159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:49.247754 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 08:21:50.247634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:50.247547 6160 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 08:21:50.247634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:50.247607 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 08:21:51.246962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:51.246927 6160 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 08:21:51.247123 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:51.246978 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 08:21:52.247737 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:52.247700 6160 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 08:21:52.248144 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:52.247759 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 08:21:52.732870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:52.732834 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:53.247389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:53.247350 6160 patch_prober.go:29] interesting pod/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 08:21:53.247595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:53.247406 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 08:21:54.327813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:54.327766 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:54.328464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:54.328440 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:54.329268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:54.329250 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:21:57.460147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:21:57.460109 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:01.412395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:01.412358 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:02.252729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:02.252681 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:02.961942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:02.961907 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:07.141899 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:07.141862 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:07.775296 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:07.775263 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:08.652260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:08.652224 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:10.521062 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:22:10.521022 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:22:11.940370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:11.940339 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:16.643449 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:16.643415 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:21.377813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:21.377774 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:23.739407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.739368 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp"] Nov 02 08:22:23.743858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.743842 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.840153 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.840116 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"kube-root-ca.crt\"" Nov 02 08:22:23.840355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.840170 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"default-dockercfg-d8n5d\"" Nov 02 08:22:23.840355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.840272 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"openshift-service-ca.crt\"" Nov 02 08:22:23.840355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.840321 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"e2e-conflict-a-3d3d3a66-kserve-self-signed-certs\"" Nov 02 08:22:23.843098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.843080 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp"] Nov 02 08:22:23.851763 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.851739 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:23.878950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.878920 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-dshm\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.879095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.878966 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfpqg\" (UniqueName: \"kubernetes.io/projected/16d1f449-c7c4-4f3f-8eb4-56816d031128-kube-api-access-tfpqg\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.879095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.879005 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-tmp-dir\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.879263 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.879120 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-model-cache\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.879263 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.879169 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-home\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.879337 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.879311 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d1f449-c7c4-4f3f-8eb4-56816d031128-tls-certs\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.980698 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.980658 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfpqg\" (UniqueName: \"kubernetes.io/projected/16d1f449-c7c4-4f3f-8eb4-56816d031128-kube-api-access-tfpqg\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.980903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.980714 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-tmp-dir\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.980903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.980765 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-model-cache\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.981012 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.980901 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-home\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.981070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.981012 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d1f449-c7c4-4f3f-8eb4-56816d031128-tls-certs\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.981136 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.981080 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-dshm\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.981213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.981155 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-tmp-dir\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.981272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.981222 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-model-cache\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.981309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.981274 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-home\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.983487 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.983460 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-dshm\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:23.983695 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:23.983677 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d1f449-c7c4-4f3f-8eb4-56816d031128-tls-certs\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:24.033742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:24.033672 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfpqg\" (UniqueName: \"kubernetes.io/projected/16d1f449-c7c4-4f3f-8eb4-56816d031128-kube-api-access-tfpqg\") pod \"e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:24.055638 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:24.055595 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:24.434038 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:24.434010 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp"] Nov 02 08:22:24.435237 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:22:24.435207 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16d1f449_c7c4_4f3f_8eb4_56816d031128.slice/crio-0a9f0dedbe1316e4538ed5dc7f57b2cb34a0ffc018c46ea67bea837f3ef25dd5 WatchSource:0}: Error finding container 0a9f0dedbe1316e4538ed5dc7f57b2cb34a0ffc018c46ea67bea837f3ef25dd5: Status 404 returned error can't find the container with id 0a9f0dedbe1316e4538ed5dc7f57b2cb34a0ffc018c46ea67bea837f3ef25dd5 Nov 02 08:22:25.091310 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:25.091270 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" event={"ID":"16d1f449-c7c4-4f3f-8eb4-56816d031128","Type":"ContainerStarted","Data":"956ab73601aed4cf6a0bd56cba0c56c8a5ba5f9e47c157271f7d1499c8fd6116"} Nov 02 08:22:25.091745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:25.091322 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" event={"ID":"16d1f449-c7c4-4f3f-8eb4-56816d031128","Type":"ContainerStarted","Data":"0a9f0dedbe1316e4538ed5dc7f57b2cb34a0ffc018c46ea67bea837f3ef25dd5"} Nov 02 08:22:25.200473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:25.200421 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" podStartSLOduration=2.200407755 podStartE2EDuration="2.200407755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:22:25.197790762 +0000 UTC m=+943.751294979" watchObservedRunningTime="2026-11-02 08:22:25.200407755 +0000 UTC m=+943.753911988" Nov 02 08:22:26.056143 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:26.056109 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:26.123489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:26.123461 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:27.522224 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:27.522164 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:30.882250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:30.882214 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:33.023434 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:33.023394 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:33.219046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:33.219018 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:34.056687 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:34.056648 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:34.069304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:34.069280 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:34.163988 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:34.163961 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:22:35.685126 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:35.685092 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:40.440799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:40.440758 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:40.521088 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:22:40.521054 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:22:45.131131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:45.131061 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:49.096879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.096836 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb"] Nov 02 08:22:49.100937 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.100920 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.146217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.146160 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.146376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.146221 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6hf\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-kube-api-access-tb6hf\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.146376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.146312 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.146376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.146368 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.146504 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.146409 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.146504 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.146486 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.146588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.146517 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe9b9f-aeae-452a-bf98-31440dba687c-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.146663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.146646 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.146719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.146678 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.149403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.149383 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-10d2fb-openshift-default-dockercfg-hh68x\"" Nov 02 08:22:49.198709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.198677 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb"] Nov 02 08:22:49.247681 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.247649 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.247681 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.247678 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.247918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.247732 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.247918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.247750 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6hf\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-kube-api-access-tb6hf\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.247918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.247768 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.247918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.247807 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.248102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.247941 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.248102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.247985 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.248102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.248027 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe9b9f-aeae-452a-bf98-31440dba687c-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.248306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.248231 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.248306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.248255 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.248410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.248351 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.248493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.248473 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.248680 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.248663 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe9b9f-aeae-452a-bf98-31440dba687c-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.250343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.250326 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.250413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.250395 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.300896 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.300859 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.301052 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.300970 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6hf\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-kube-api-access-tb6hf\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.414140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.414105 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:49.837268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.837239 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb"] Nov 02 08:22:49.837578 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:22:49.837557 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fe9b9f_aeae_452a_bf98_31440dba687c.slice/crio-13ea2c6de1c938cd1d043322129d3f288c97d385ae69f4eaf42991f86a1581a1 WatchSource:0}: Error finding container 13ea2c6de1c938cd1d043322129d3f288c97d385ae69f4eaf42991f86a1581a1: Status 404 returned error can't find the container with id 13ea2c6de1c938cd1d043322129d3f288c97d385ae69f4eaf42991f86a1581a1 Nov 02 08:22:49.839824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.839791 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:22:49.839905 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.839870 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:22:49.839950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.839912 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:22:49.978020 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:49.977936 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:50.260946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:50.260905 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" event={"ID":"04fe9b9f-aeae-452a-bf98-31440dba687c","Type":"ContainerStarted","Data":"6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5"} Nov 02 08:22:50.261480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:50.261013 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" event={"ID":"04fe9b9f-aeae-452a-bf98-31440dba687c","Type":"ContainerStarted","Data":"13ea2c6de1c938cd1d043322129d3f288c97d385ae69f4eaf42991f86a1581a1"} Nov 02 08:22:50.377956 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:50.377897 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podStartSLOduration=1.377880605 podStartE2EDuration="1.377880605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:22:50.375824335 +0000 UTC m=+968.929328552" watchObservedRunningTime="2026-11-02 08:22:50.377880605 +0000 UTC m=+968.931384794" Nov 02 08:22:50.431419 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:50.431380 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:50.433217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:50.433159 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 08:22:50.433327 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:50.433244 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 08:22:51.420663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:51.420622 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 08:22:51.421330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:51.421297 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 08:22:52.420211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:52.418552 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 08:22:52.420211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:52.418613 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 08:22:53.415607 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:53.415563 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 08:22:53.416225 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:53.416167 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 08:22:54.423215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:54.422614 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 08:22:54.423215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:54.422679 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 08:22:54.832696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:54.832595 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:55.359497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:55.359463 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:55.427203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:55.425613 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 08:22:55.427203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:55.425678 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 08:22:56.415343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:56.415299 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 08:22:56.415517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:56.415382 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 08:22:57.419081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:57.419047 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:57.419591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:57.419500 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:57.420169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:57.420152 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:22:59.640137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:22:59.640104 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:03.319133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:03.319052 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:04.529340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:04.529292 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:05.557576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:05.557546 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:08.110398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:08.110367 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:09.347602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:09.347562 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:10.521265 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:23:10.521222 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:23:14.033468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:14.033435 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:18.846032 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:18.845996 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:19.014395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:19.014364 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:23.678106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:23.678008 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:27.856307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:27.856264 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:28.526980 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:28.526940 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:33.228955 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:33.228917 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:34.008165 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.008130 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn"] Nov 02 08:23:34.012256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.012239 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.061453 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.061415 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8e6407-openshift-default-dockercfg-lcnm5\"" Nov 02 08:23:34.110819 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.110782 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn"] Nov 02 08:23:34.142063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.142030 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.142260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.142073 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.142260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.142111 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knrrn\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-kube-api-access-knrrn\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.142260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.142213 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.142377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.142299 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.142377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.142343 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.142443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.142421 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.142479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.142469 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.142512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.142487 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244038 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244073 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244111 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244134 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244167 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knrrn\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-kube-api-access-knrrn\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244231 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244296 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244355 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244394 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244675 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244727 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244789 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.244948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.244863 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.245414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.245389 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.246692 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.246674 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.246761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.246745 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.297838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.297769 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.297838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.297793 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knrrn\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-kube-api-access-knrrn\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.326184 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.326158 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:34.746592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.746565 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn"] Nov 02 08:23:34.747913 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:23:34.747887 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e44d29d_9365_478a_ba4b_33b5e1b7f9ac.slice/crio-c2044c5b033da28bd681470df59ccd7455f0e95e91fcc5e37df43b468bfd2894 WatchSource:0}: Error finding container c2044c5b033da28bd681470df59ccd7455f0e95e91fcc5e37df43b468bfd2894: Status 404 returned error can't find the container with id c2044c5b033da28bd681470df59ccd7455f0e95e91fcc5e37df43b468bfd2894 Nov 02 08:23:34.750319 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.750284 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:23:34.750394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.750349 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:23:34.750394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:34.750382 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:23:35.545468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:35.545430 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" event={"ID":"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac","Type":"ContainerStarted","Data":"4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e"} Nov 02 08:23:35.545978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:35.545496 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" event={"ID":"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac","Type":"ContainerStarted","Data":"c2044c5b033da28bd681470df59ccd7455f0e95e91fcc5e37df43b468bfd2894"} Nov 02 08:23:35.701271 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:35.701169 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" podStartSLOduration=2.701147334 podStartE2EDuration="2.701147334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:23:35.699846339 +0000 UTC m=+1014.253350536" watchObservedRunningTime="2026-11-02 08:23:35.701147334 +0000 UTC m=+1014.254651532" Nov 02 08:23:36.331463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:36.331403 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:36.333783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:36.333748 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 08:23:36.333943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:36.333847 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 08:23:37.331682 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:37.331644 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 08:23:37.332082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:37.331721 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 08:23:38.026202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:38.025755 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:38.329257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:38.328464 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 08:23:38.329257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:38.328562 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 08:23:39.330645 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:39.330606 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 08:23:39.331275 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:39.331230 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 08:23:40.326754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:40.326713 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 08:23:40.327075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:40.327034 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 08:23:40.545639 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:23:40.545605 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:23:41.497282 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:41.497237 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:41.497695 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:41.497632 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:41.498908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:41.498890 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:23:42.804797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:42.804764 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:47.547292 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:47.547243 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:52.260871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:52.260837 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:57.044030 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:57.043993 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:58.963073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:23:58.963034 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:01.882292 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:01.882255 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:03.637251 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:03.637214 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:06.734828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:06.734784 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:08.447638 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:08.447598 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:10.522211 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:24:10.522159 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:24:11.532997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:11.532961 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:16.331346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:16.331308 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:21.135917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:21.135883 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:25.321562 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.321523 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn"] Nov 02 08:24:25.325492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.325474 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.420764 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.420731 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55vx\" (UniqueName: \"kubernetes.io/projected/620891d5-0794-4427-b70c-6d8ee3a0c534-kube-api-access-w55vx\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.420941 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.420794 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-home\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.420941 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.420849 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/620891d5-0794-4427-b70c-6d8ee3a0c534-tls-certs\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.421019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.420960 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-model-cache\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.421019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.421001 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-dshm\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.421081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.421025 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-tmp-dir\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.422938 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.422920 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"e2e-conflict-b-3d3d3a66-kserve-self-signed-certs\"" Nov 02 08:24:25.425778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.425757 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn"] Nov 02 08:24:25.440248 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.440220 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:25.522515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.522484 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-model-cache\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.522697 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.522538 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-dshm\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.522697 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.522564 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-tmp-dir\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.522776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.522734 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55vx\" (UniqueName: \"kubernetes.io/projected/620891d5-0794-4427-b70c-6d8ee3a0c534-kube-api-access-w55vx\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.522819 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.522801 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-home\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.522856 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.522820 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/620891d5-0794-4427-b70c-6d8ee3a0c534-tls-certs\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.522905 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.522857 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-model-cache\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.522972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.522945 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-tmp-dir\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.523245 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.523224 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-home\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.525008 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.524986 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-dshm\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.525526 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.525508 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/620891d5-0794-4427-b70c-6d8ee3a0c534-tls-certs\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.575939 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.575873 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55vx\" (UniqueName: \"kubernetes.io/projected/620891d5-0794-4427-b70c-6d8ee3a0c534-kube-api-access-w55vx\") pod \"e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.637767 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.637736 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:25.942761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:25.942724 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:26.023807 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:26.023783 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn"] Nov 02 08:24:26.025057 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:24:26.025024 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod620891d5_0794_4427_b70c_6d8ee3a0c534.slice/crio-817428d5d1b277d3ba66ff8df9d7fde05b80c225e70b83b3be0100bc6ce3c0be WatchSource:0}: Error finding container 817428d5d1b277d3ba66ff8df9d7fde05b80c225e70b83b3be0100bc6ce3c0be: Status 404 returned error can't find the container with id 817428d5d1b277d3ba66ff8df9d7fde05b80c225e70b83b3be0100bc6ce3c0be Nov 02 08:24:26.882148 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:26.882110 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" event={"ID":"620891d5-0794-4427-b70c-6d8ee3a0c534","Type":"ContainerStarted","Data":"c4bdede9ea68726629415343ace1bc8717ca215602fd9ebedd4028d6af44a557"} Nov 02 08:24:26.882544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:26.882158 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" event={"ID":"620891d5-0794-4427-b70c-6d8ee3a0c534","Type":"ContainerStarted","Data":"817428d5d1b277d3ba66ff8df9d7fde05b80c225e70b83b3be0100bc6ce3c0be"} Nov 02 08:24:26.991657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:26.991598 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" podStartSLOduration=1.991584178 podStartE2EDuration="1.991584178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:26.987800837 +0000 UTC m=+1065.541305034" watchObservedRunningTime="2026-11-02 08:24:26.991584178 +0000 UTC m=+1065.545088374" Nov 02 08:24:27.638468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:27.638430 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:28.177341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:28.177299 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:30.485418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:30.485384 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:30.745303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:30.745216 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:35.538605 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:35.538570 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:35.638926 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:35.638889 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:35.652618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:35.652583 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:35.951715 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:35.951688 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:24:40.324037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:40.324002 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:40.525030 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:24:40.524994 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:24:45.175036 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:45.174995 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:45.356061 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:45.356018 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:46.989298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:46.989257 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:47.148039 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:47.147997 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:49.940924 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:49.940888 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:54.718786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:54.718742 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:59.279827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:59.279790 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:59.440814 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:24:59.440775 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:02.168671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.168635 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn"] Nov 02 08:25:02.172985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.172963 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.221685 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.221647 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-096685-openshift-default-dockercfg-fcvv8\"" Nov 02 08:25:02.228028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.228003 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-podinfo\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.228121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.228042 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-data\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.228217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.228192 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-credential-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.228264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.228248 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-token\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.228298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.228282 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be98284b-e24f-4a50-bef2-c37d8f7dd924-istiod-ca-cert\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.228332 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.228308 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.228440 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.228421 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9l46\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-kube-api-access-g9l46\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.228507 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.228495 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-envoy\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.228543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.228521 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-certs\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.271260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.271220 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn"] Nov 02 08:25:02.329270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329231 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329294 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9l46\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-kube-api-access-g9l46\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329352 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-envoy\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329377 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-certs\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329570 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-podinfo\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329613 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-data\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329661 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-credential-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329672 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329808 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329720 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-token\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329808 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329727 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-certs\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329808 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329791 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be98284b-e24f-4a50-bef2-c37d8f7dd924-istiod-ca-cert\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.329969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329939 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-data\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.330025 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.329966 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-credential-socket\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.330446 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.330425 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be98284b-e24f-4a50-bef2-c37d8f7dd924-istiod-ca-cert\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.331830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.331803 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-envoy\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.332080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.332063 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-podinfo\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.382497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.382467 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-token\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.382642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.382565 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9l46\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-kube-api-access-g9l46\") pod \"e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.487972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.487874 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:02.907750 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.907717 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn"] Nov 02 08:25:02.909426 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:25:02.909389 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe98284b_e24f_4a50_bef2_c37d8f7dd924.slice/crio-080509ad96a238a0ef8c8698e648b33a018fca1e0ff5295e73a662cf4ebeef53 WatchSource:0}: Error finding container 080509ad96a238a0ef8c8698e648b33a018fca1e0ff5295e73a662cf4ebeef53: Status 404 returned error can't find the container with id 080509ad96a238a0ef8c8698e648b33a018fca1e0ff5295e73a662cf4ebeef53 Nov 02 08:25:02.911850 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.911814 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:25:02.911947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.911897 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:25:02.911947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:02.911941 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:25:03.115087 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:03.115046 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" event={"ID":"be98284b-e24f-4a50-bef2-c37d8f7dd924","Type":"ContainerStarted","Data":"7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f"} Nov 02 08:25:03.115315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:03.115116 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" event={"ID":"be98284b-e24f-4a50-bef2-c37d8f7dd924","Type":"ContainerStarted","Data":"080509ad96a238a0ef8c8698e648b33a018fca1e0ff5295e73a662cf4ebeef53"} Nov 02 08:25:03.227021 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:03.226951 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" podStartSLOduration=1.2269316909999999 podStartE2EDuration="1.226931691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:25:03.225488145 +0000 UTC m=+1101.778992363" watchObservedRunningTime="2026-11-02 08:25:03.226931691 +0000 UTC m=+1101.780435889" Nov 02 08:25:03.488665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:03.488571 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:03.489792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:03.489766 6160 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 08:25:03.489858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:03.489828 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 08:25:03.994237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:03.994193 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:04.275273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:04.274338 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:04.489163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:04.489127 6160 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 08:25:04.489349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:04.489217 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 08:25:05.488671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:05.488637 6160 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 08:25:05.489074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:05.488691 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 08:25:06.489008 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:06.488969 6160 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 08:25:06.489433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:06.489024 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 08:25:07.488881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:07.488847 6160 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 08:25:07.489069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:07.488900 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 08:25:08.488839 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:08.488804 6160 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 08:25:08.489024 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:08.488872 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 08:25:08.807444 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:08.807352 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:09.076439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:09.076331 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:09.489265 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:09.489229 6160 patch_prober.go:29] interesting pod/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 08:25:09.489442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:09.489290 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 08:25:10.493250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:10.493221 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:10.493749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:10.493724 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:10.494452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:10.494432 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:25:10.520748 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:25:10.520713 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:25:13.793592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:13.793554 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:13.994244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:13.994210 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:18.743395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:18.743358 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:23.524299 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:23.524263 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:28.279960 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:28.279919 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:33.038339 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:33.038300 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:37.844991 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:37.844951 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:40.521430 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:25:40.521393 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:25:42.568406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:42.568365 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:44.397565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.397530 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb"] Nov 02 08:25:44.406709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.406673 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.456738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.456704 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb"] Nov 02 08:25:44.476886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.476853 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:44.499219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.499189 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e6407\"/\"distinct-w4-8e6407-kserve-self-signed-certs\"" Nov 02 08:25:44.548523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.548484 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tls-certs\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.548896 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.548863 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-dshm\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.549131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.549111 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-model-cache\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.549596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.549578 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-home\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.549844 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.549826 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmc7x\" (UniqueName: \"kubernetes.io/projected/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-kube-api-access-pmc7x\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.550265 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.550245 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tmp-dir\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.652249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.652131 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-home\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.652249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.652225 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc7x\" (UniqueName: \"kubernetes.io/projected/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-kube-api-access-pmc7x\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.652476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.652292 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tmp-dir\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.652476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.652376 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tls-certs\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.652476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.652413 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-dshm\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.652476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.652451 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-model-cache\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.652756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.652610 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-home\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.652756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.652707 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tmp-dir\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.652861 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.652801 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-model-cache\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.654958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.654935 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-dshm\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.655298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.655282 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tls-certs\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.711373 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.711339 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmc7x\" (UniqueName: \"kubernetes.io/projected/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-kube-api-access-pmc7x\") pod \"distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:44.719288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:44.719267 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:45.120759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:45.120736 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb"] Nov 02 08:25:45.122589 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:25:45.122553 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a1f3909_2d7e_4382_9f2d_c9dbdeb0cfde.slice/crio-94c00c5fbd9dbbe70280f699c8343f756c2e1faf0974e9c03bfe43f5091a0093 WatchSource:0}: Error finding container 94c00c5fbd9dbbe70280f699c8343f756c2e1faf0974e9c03bfe43f5091a0093: Status 404 returned error can't find the container with id 94c00c5fbd9dbbe70280f699c8343f756c2e1faf0974e9c03bfe43f5091a0093 Nov 02 08:25:45.391600 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:45.391511 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" event={"ID":"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde","Type":"ContainerStarted","Data":"394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370"} Nov 02 08:25:45.391600 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:45.391558 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" event={"ID":"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde","Type":"ContainerStarted","Data":"94c00c5fbd9dbbe70280f699c8343f756c2e1faf0974e9c03bfe43f5091a0093"} Nov 02 08:25:45.501515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:45.501446 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" podStartSLOduration=1.5014322820000001 podStartE2EDuration="1.501432282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:25:45.498738878 +0000 UTC m=+1144.052243095" watchObservedRunningTime="2026-11-02 08:25:45.501432282 +0000 UTC m=+1144.054936479" Nov 02 08:25:45.719692 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:45.719661 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:47.345384 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:47.345341 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:52.077875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:52.077831 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:54.720019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:54.719977 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:54.732923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:54.732894 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:55.466830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:55.466798 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:25:56.772684 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:25:56.772651 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:01.533645 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:01.533602 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:04.320373 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:04.320341 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:06.270954 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:06.270891 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:09.134195 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:09.134145 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:09.812418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:09.812378 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:10.521398 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:26:10.521361 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:26:11.124957 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:11.124920 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:15.920950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:15.920917 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:20.707904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:20.707822 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:25.448983 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:25.448938 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:25.559884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:25.559856 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn"] Nov 02 08:26:25.646627 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:25.646564 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:30.489100 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.489061 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:30.673859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.673826 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv"] Nov 02 08:26:30.677977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.677959 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.752197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.752074 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d0628e\"/\"openshift-service-ca.crt\"" Nov 02 08:26:30.759801 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.759772 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-tmp-dir\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.759931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.759883 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzlq\" (UniqueName: \"kubernetes.io/projected/78052f88-4743-4746-98ec-8f0ecb9d21cd-kube-api-access-zlzlq\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.759978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.759938 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-model-cache\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.759978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.759960 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-dshm\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.760065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.760035 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78052f88-4743-4746-98ec-8f0ecb9d21cd-tls-certs\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.760203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.760165 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-home\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.775248 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.775220 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d0628e\"/\"distinct-2-w1-d0628e-kserve-self-signed-certs\"" Nov 02 08:26:30.775376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.775248 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d0628e\"/\"kube-root-ca.crt\"" Nov 02 08:26:30.775376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.775331 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d0628e\"/\"default-dockercfg-g4f7x\"" Nov 02 08:26:30.778110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.778082 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv"] Nov 02 08:26:30.800915 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.800879 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:30.860871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.860841 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-home\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.861083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.860892 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-tmp-dir\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.861083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.860947 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzlq\" (UniqueName: \"kubernetes.io/projected/78052f88-4743-4746-98ec-8f0ecb9d21cd-kube-api-access-zlzlq\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.861083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.860990 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-model-cache\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.861083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.861021 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-dshm\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.861083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.861053 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78052f88-4743-4746-98ec-8f0ecb9d21cd-tls-certs\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.861384 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.861369 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-home\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.861443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.861414 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-tmp-dir\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.861502 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.861472 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-model-cache\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.863821 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.863793 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-dshm\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.863939 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.863802 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78052f88-4743-4746-98ec-8f0ecb9d21cd-tls-certs\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.915408 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.915383 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzlq\" (UniqueName: \"kubernetes.io/projected/78052f88-4743-4746-98ec-8f0ecb9d21cd-kube-api-access-zlzlq\") pod \"distinct-2-w1-d0628e-kserve-56d664698d-klwzv\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:30.991793 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:30.991767 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:31.378244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:31.378212 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv"] Nov 02 08:26:31.379376 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:26:31.379347 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78052f88_4743_4746_98ec_8f0ecb9d21cd.slice/crio-49b76842e63aa2cc859e22fa03d1cf0ef600d83adc1bc2d7eeccaf48aa7cece8 WatchSource:0}: Error finding container 49b76842e63aa2cc859e22fa03d1cf0ef600d83adc1bc2d7eeccaf48aa7cece8: Status 404 returned error can't find the container with id 49b76842e63aa2cc859e22fa03d1cf0ef600d83adc1bc2d7eeccaf48aa7cece8 Nov 02 08:26:31.699846 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:31.699810 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" event={"ID":"78052f88-4743-4746-98ec-8f0ecb9d21cd","Type":"ContainerStarted","Data":"770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850"} Nov 02 08:26:31.699846 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:31.699851 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" event={"ID":"78052f88-4743-4746-98ec-8f0ecb9d21cd","Type":"ContainerStarted","Data":"49b76842e63aa2cc859e22fa03d1cf0ef600d83adc1bc2d7eeccaf48aa7cece8"} Nov 02 08:26:31.806803 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:31.806746 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" podStartSLOduration=1.806729845 podStartE2EDuration="1.806729845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:26:31.806352994 +0000 UTC m=+1190.359857201" watchObservedRunningTime="2026-11-02 08:26:31.806729845 +0000 UTC m=+1190.360234052" Nov 02 08:26:31.991987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:31.991907 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:33.521298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:33.521258 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:35.248326 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:35.248288 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:35.624711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:35.624625 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp"] Nov 02 08:26:35.665391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:35.665356 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:40.143936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.143900 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:40.520940 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:26:40.520904 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:26:40.580511 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.580474 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" podUID="620891d5-0794-4427-b70c-6d8ee3a0c534" containerName="main" containerID="cri-o://c4bdede9ea68726629415343ace1bc8717ca215602fd9ebedd4028d6af44a557" gracePeriod=45 Nov 02 08:26:40.767410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.767376 6160 generic.go:356] "Generic (PLEG): container finished" podID="620891d5-0794-4427-b70c-6d8ee3a0c534" containerID="c4bdede9ea68726629415343ace1bc8717ca215602fd9ebedd4028d6af44a557" exitCode=0 Nov 02 08:26:40.767563 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.767513 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" event={"ID":"620891d5-0794-4427-b70c-6d8ee3a0c534","Type":"ContainerDied","Data":"c4bdede9ea68726629415343ace1bc8717ca215602fd9ebedd4028d6af44a557"} Nov 02 08:26:40.837962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.837936 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:26:40.974585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.974546 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-model-cache\") pod \"620891d5-0794-4427-b70c-6d8ee3a0c534\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " Nov 02 08:26:40.974766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.974748 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-tmp-dir\") pod \"620891d5-0794-4427-b70c-6d8ee3a0c534\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " Nov 02 08:26:40.974835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.974793 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-dshm\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-dshm\") pod \"620891d5-0794-4427-b70c-6d8ee3a0c534\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " Nov 02 08:26:40.974891 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.974834 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620891d5-0794-4427-b70c-6d8ee3a0c534-kube-api-access-w55vx\" (UniqueName: \"kubernetes.io/projected/620891d5-0794-4427-b70c-6d8ee3a0c534-kube-api-access-w55vx\") pod \"620891d5-0794-4427-b70c-6d8ee3a0c534\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " Nov 02 08:26:40.974953 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.974891 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-home\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-home\") pod \"620891d5-0794-4427-b70c-6d8ee3a0c534\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " Nov 02 08:26:40.974953 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.974918 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/620891d5-0794-4427-b70c-6d8ee3a0c534-tls-certs\" (UniqueName: \"kubernetes.io/secret/620891d5-0794-4427-b70c-6d8ee3a0c534-tls-certs\") pod \"620891d5-0794-4427-b70c-6d8ee3a0c534\" (UID: \"620891d5-0794-4427-b70c-6d8ee3a0c534\") " Nov 02 08:26:40.975057 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.974980 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-model-cache" pod "620891d5-0794-4427-b70c-6d8ee3a0c534" (UID: "620891d5-0794-4427-b70c-6d8ee3a0c534"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:40.975057 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.975037 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-tmp-dir" pod "620891d5-0794-4427-b70c-6d8ee3a0c534" (UID: "620891d5-0794-4427-b70c-6d8ee3a0c534"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:40.975389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.975361 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-home" pod "620891d5-0794-4427-b70c-6d8ee3a0c534" (UID: "620891d5-0794-4427-b70c-6d8ee3a0c534"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:40.975539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.975482 6160 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-model-cache\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:40.975539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.975502 6160 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-tmp-dir\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:40.977605 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.977579 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-dshm" pod "620891d5-0794-4427-b70c-6d8ee3a0c534" (UID: "620891d5-0794-4427-b70c-6d8ee3a0c534"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:40.977746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.977709 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620891d5-0794-4427-b70c-6d8ee3a0c534-kube-api-access-w55vx" pod "620891d5-0794-4427-b70c-6d8ee3a0c534" (UID: "620891d5-0794-4427-b70c-6d8ee3a0c534"). InnerVolumeSpecName "kube-api-access-w55vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:40.984390 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.984358 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/620891d5-0794-4427-b70c-6d8ee3a0c534-tls-certs" pod "620891d5-0794-4427-b70c-6d8ee3a0c534" (UID: "620891d5-0794-4427-b70c-6d8ee3a0c534"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:40.992271 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:40.992247 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:41.004981 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.004962 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:41.076478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.076381 6160 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-dshm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:41.076478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.076414 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55vx\" (UniqueName: \"kubernetes.io/projected/620891d5-0794-4427-b70c-6d8ee3a0c534-kube-api-access-w55vx\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:41.076478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.076429 6160 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/620891d5-0794-4427-b70c-6d8ee3a0c534-home\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:41.076478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.076441 6160 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/620891d5-0794-4427-b70c-6d8ee3a0c534-tls-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:41.751621 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.751592 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:26:41.752361 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.752341 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:26:41.775349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.775319 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" Nov 02 08:26:41.775349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.775332 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn" event={"ID":"620891d5-0794-4427-b70c-6d8ee3a0c534","Type":"ContainerDied","Data":"817428d5d1b277d3ba66ff8df9d7fde05b80c225e70b83b3be0100bc6ce3c0be"} Nov 02 08:26:41.775585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.775377 6160 scope.go:122] "RemoveContainer" containerID="c4bdede9ea68726629415343ace1bc8717ca215602fd9ebedd4028d6af44a557" Nov 02 08:26:41.787717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.787695 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:26:41.928389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.928354 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn"] Nov 02 08:26:41.933727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:41.933703 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-b-3d3d3a66-kserve-c84d459df-lcmfn"] Nov 02 08:26:43.585541 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:43.585505 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620891d5-0794-4427-b70c-6d8ee3a0c534" path="/var/lib/kubelet/pods/620891d5-0794-4427-b70c-6d8ee3a0c534/volumes" Nov 02 08:26:44.836113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:44.836074 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:49.646308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:49.646272 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:50.645672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:50.645627 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" podUID="16d1f449-c7c4-4f3f-8eb4-56816d031128" containerName="main" containerID="cri-o://956ab73601aed4cf6a0bd56cba0c56c8a5ba5f9e47c157271f7d1499c8fd6116" gracePeriod=45 Nov 02 08:26:50.855382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:50.855343 6160 generic.go:356] "Generic (PLEG): container finished" podID="16d1f449-c7c4-4f3f-8eb4-56816d031128" containerID="956ab73601aed4cf6a0bd56cba0c56c8a5ba5f9e47c157271f7d1499c8fd6116" exitCode=0 Nov 02 08:26:50.855735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:50.855425 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" event={"ID":"16d1f449-c7c4-4f3f-8eb4-56816d031128","Type":"ContainerDied","Data":"956ab73601aed4cf6a0bd56cba0c56c8a5ba5f9e47c157271f7d1499c8fd6116"} Nov 02 08:26:50.922729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:50.922700 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:26:51.084511 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.084480 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-dshm\") pod \"16d1f449-c7c4-4f3f-8eb4-56816d031128\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " Nov 02 08:26:51.084680 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.084532 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d1f449-c7c4-4f3f-8eb4-56816d031128-kube-api-access-tfpqg\" (UniqueName: \"kubernetes.io/projected/16d1f449-c7c4-4f3f-8eb4-56816d031128-kube-api-access-tfpqg\") pod \"16d1f449-c7c4-4f3f-8eb4-56816d031128\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " Nov 02 08:26:51.084680 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.084595 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-home\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-home\") pod \"16d1f449-c7c4-4f3f-8eb4-56816d031128\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " Nov 02 08:26:51.084680 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.084638 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d1f449-c7c4-4f3f-8eb4-56816d031128-tls-certs\" (UniqueName: \"kubernetes.io/secret/16d1f449-c7c4-4f3f-8eb4-56816d031128-tls-certs\") pod \"16d1f449-c7c4-4f3f-8eb4-56816d031128\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " Nov 02 08:26:51.084680 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.084668 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-model-cache\") pod \"16d1f449-c7c4-4f3f-8eb4-56816d031128\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " Nov 02 08:26:51.084823 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.084788 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-tmp-dir\") pod \"16d1f449-c7c4-4f3f-8eb4-56816d031128\" (UID: \"16d1f449-c7c4-4f3f-8eb4-56816d031128\") " Nov 02 08:26:51.084975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.084942 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-home" pod "16d1f449-c7c4-4f3f-8eb4-56816d031128" (UID: "16d1f449-c7c4-4f3f-8eb4-56816d031128"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:51.085136 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.084971 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-model-cache" pod "16d1f449-c7c4-4f3f-8eb4-56816d031128" (UID: "16d1f449-c7c4-4f3f-8eb4-56816d031128"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:51.085136 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.085076 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-tmp-dir" pod "16d1f449-c7c4-4f3f-8eb4-56816d031128" (UID: "16d1f449-c7c4-4f3f-8eb4-56816d031128"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:51.085524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.085505 6160 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-home\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:51.085570 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.085532 6160 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-model-cache\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:51.085610 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.085575 6160 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-tmp-dir\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:51.086974 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.086954 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-dshm" pod "16d1f449-c7c4-4f3f-8eb4-56816d031128" (UID: "16d1f449-c7c4-4f3f-8eb4-56816d031128"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:51.087035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.086968 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d1f449-c7c4-4f3f-8eb4-56816d031128-tls-certs" pod "16d1f449-c7c4-4f3f-8eb4-56816d031128" (UID: "16d1f449-c7c4-4f3f-8eb4-56816d031128"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:51.087035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.087007 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d1f449-c7c4-4f3f-8eb4-56816d031128-kube-api-access-tfpqg" pod "16d1f449-c7c4-4f3f-8eb4-56816d031128" (UID: "16d1f449-c7c4-4f3f-8eb4-56816d031128"). InnerVolumeSpecName "kube-api-access-tfpqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:51.187049 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.187014 6160 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d1f449-c7c4-4f3f-8eb4-56816d031128-dshm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:51.187049 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.187047 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfpqg\" (UniqueName: \"kubernetes.io/projected/16d1f449-c7c4-4f3f-8eb4-56816d031128-kube-api-access-tfpqg\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:51.187289 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.187059 6160 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d1f449-c7c4-4f3f-8eb4-56816d031128-tls-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:26:51.862540 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.862462 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" Nov 02 08:26:51.862540 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.862476 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp" event={"ID":"16d1f449-c7c4-4f3f-8eb4-56816d031128","Type":"ContainerDied","Data":"0a9f0dedbe1316e4538ed5dc7f57b2cb34a0ffc018c46ea67bea837f3ef25dd5"} Nov 02 08:26:51.862540 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:51.862525 6160 scope.go:122] "RemoveContainer" containerID="956ab73601aed4cf6a0bd56cba0c56c8a5ba5f9e47c157271f7d1499c8fd6116" Nov 02 08:26:52.015949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:52.015909 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp"] Nov 02 08:26:52.061701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:52.061670 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/e2e-conflict-a-3d3d3a66-kserve-cbf855759-dg2tp"] Nov 02 08:26:53.584641 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:53.584604 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d1f449-c7c4-4f3f-8eb4-56816d031128" path="/var/lib/kubelet/pods/16d1f449-c7c4-4f3f-8eb4-56816d031128/volumes" Nov 02 08:26:54.430751 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:54.430707 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:55.192639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:55.192606 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn"] Nov 02 08:26:55.193059 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:55.192836 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" containerID="cri-o://9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a" gracePeriod=30 Nov 02 08:26:59.264417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:59.264382 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:59.729942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:59.729904 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8t2kn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:26:59.730142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:26:59.729980 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:27:00.457582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.457516 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:27:00.491484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491450 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istiod-ca-cert\") pod \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " Nov 02 08:27:00.491484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491488 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-socket\") pod \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " Nov 02 08:27:00.491710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491539 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-certs\") pod \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " Nov 02 08:27:00.491710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491560 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-kube-api-access-2ml2x\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-kube-api-access-2ml2x\") pod \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " Nov 02 08:27:00.491710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491591 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-data\") pod \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " Nov 02 08:27:00.491710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491614 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-token\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-token\") pod \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " Nov 02 08:27:00.491710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491691 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-podinfo\") pod \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " Nov 02 08:27:00.491972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491743 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-envoy\") pod \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " Nov 02 08:27:00.491972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491815 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-credential-socket\") pod \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\" (UID: \"6d462b83-c9b7-4cba-8aed-16fba5fffa52\") " Nov 02 08:27:00.491972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491870 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-certs" pod "6d462b83-c9b7-4cba-8aed-16fba5fffa52" (UID: "6d462b83-c9b7-4cba-8aed-16fba5fffa52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:27:00.491972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.491891 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istiod-ca-cert" pod "6d462b83-c9b7-4cba-8aed-16fba5fffa52" (UID: "6d462b83-c9b7-4cba-8aed-16fba5fffa52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:27:00.492169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.492000 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-data" pod "6d462b83-c9b7-4cba-8aed-16fba5fffa52" (UID: "6d462b83-c9b7-4cba-8aed-16fba5fffa52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:27:00.492169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.492116 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-socket" pod "6d462b83-c9b7-4cba-8aed-16fba5fffa52" (UID: "6d462b83-c9b7-4cba-8aed-16fba5fffa52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:27:00.492288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.492249 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-credential-socket" pod "6d462b83-c9b7-4cba-8aed-16fba5fffa52" (UID: "6d462b83-c9b7-4cba-8aed-16fba5fffa52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:27:00.492565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.492478 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:00.492565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.492564 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:00.492768 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.492579 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:00.492768 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.492593 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:00.492768 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.492606 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:00.494081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.494049 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-podinfo" pod "6d462b83-c9b7-4cba-8aed-16fba5fffa52" (UID: "6d462b83-c9b7-4cba-8aed-16fba5fffa52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:27:00.494081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.494066 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-kube-api-access-2ml2x" pod "6d462b83-c9b7-4cba-8aed-16fba5fffa52" (UID: "6d462b83-c9b7-4cba-8aed-16fba5fffa52"). InnerVolumeSpecName "kube-api-access-2ml2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:27:00.494081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.494050 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-token" pod "6d462b83-c9b7-4cba-8aed-16fba5fffa52" (UID: "6d462b83-c9b7-4cba-8aed-16fba5fffa52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:27:00.494281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.494159 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-envoy" pod "6d462b83-c9b7-4cba-8aed-16fba5fffa52" (UID: "6d462b83-c9b7-4cba-8aed-16fba5fffa52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:27:00.593833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.593803 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ml2x\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-kube-api-access-2ml2x\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:00.593833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.593834 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:00.594035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.593848 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:00.594035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.593860 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d462b83-c9b7-4cba-8aed-16fba5fffa52-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:27:00.925401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.925368 6160 generic.go:356] "Generic (PLEG): container finished" podID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerID="9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a" exitCode=0 Nov 02 08:27:00.925575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.925413 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" Nov 02 08:27:00.925575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.925449 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" event={"ID":"6d462b83-c9b7-4cba-8aed-16fba5fffa52","Type":"ContainerDied","Data":"9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a"} Nov 02 08:27:00.925575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.925507 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn" event={"ID":"6d462b83-c9b7-4cba-8aed-16fba5fffa52","Type":"ContainerDied","Data":"40f534665b758b424bf1a1ee45c8088cf55b6147c1fb01ce98df704a9447c5cf"} Nov 02 08:27:00.925575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.925532 6160 scope.go:122] "RemoveContainer" containerID="9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a" Nov 02 08:27:00.936538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.936521 6160 scope.go:122] "RemoveContainer" containerID="9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a" Nov 02 08:27:00.936819 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:27:00.936800 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a\": container with ID starting with 9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a not found: ID does not exist" containerID="9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a" Nov 02 08:27:00.936866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:00.936827 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a"} err="failed to get container status \"9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a\": rpc error: code = NotFound desc = could not find container \"9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a\": container with ID starting with 9677eb1d739cb9cdb9276ad4a7416f62bb5e2d7ce830d4ab1ad879ab5503de6a not found: ID does not exist" Nov 02 08:27:01.080625 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:01.080583 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn"] Nov 02 08:27:01.126968 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:01.126927 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8t2kn"] Nov 02 08:27:01.584585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:01.584554 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" path="/var/lib/kubelet/pods/6d462b83-c9b7-4cba-8aed-16fba5fffa52/volumes" Nov 02 08:27:04.096370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:04.096330 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:04.657089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:04.657052 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:08.175933 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:08.175890 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:08.843712 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:08.843677 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:09.465575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:09.465535 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:10.522592 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:27:10.522546 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:27:13.521471 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.521430 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw"] Nov 02 08:27:13.522017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.522003 6160 cpu_manager.go:414] "containerMap: removing container" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" Nov 02 08:27:13.522017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.522019 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" Nov 02 08:27:13.522104 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.522052 6160 cpu_manager.go:414] "containerMap: removing container" podUID="620891d5-0794-4427-b70c-6d8ee3a0c534" containerName="main" Nov 02 08:27:13.522104 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.522057 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620891d5-0794-4427-b70c-6d8ee3a0c534" containerName="main" Nov 02 08:27:13.522104 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.522070 6160 cpu_manager.go:414] "containerMap: removing container" podUID="16d1f449-c7c4-4f3f-8eb4-56816d031128" containerName="main" Nov 02 08:27:13.522104 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.522076 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d1f449-c7c4-4f3f-8eb4-56816d031128" containerName="main" Nov 02 08:27:13.522246 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.522156 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="620891d5-0794-4427-b70c-6d8ee3a0c534" containerName="main" Nov 02 08:27:13.522246 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.522164 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d1f449-c7c4-4f3f-8eb4-56816d031128" containerName="main" Nov 02 08:27:13.522246 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.522189 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d462b83-c9b7-4cba-8aed-16fba5fffa52" containerName="istio-proxy" Nov 02 08:27:13.527143 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.527123 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.624365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.624297 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw"] Nov 02 08:27:13.640371 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.640335 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946115c9-5d90-431d-8213-9724c9f1a430-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.640566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.640388 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946115c9-5d90-431d-8213-9724c9f1a430-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.640566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.640515 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.640566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.640550 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.640747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.640571 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.640747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.640596 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.640747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.640696 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.640747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.640740 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.640959 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.640764 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxwm\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-kube-api-access-tjxwm\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.646750 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.646719 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:13.741883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.741842 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.741914 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.741946 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxwm\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-kube-api-access-tjxwm\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742018 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946115c9-5d90-431d-8213-9724c9f1a430-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742059 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946115c9-5d90-431d-8213-9724c9f1a430-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742109 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742146 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742202 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742310 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742556 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742424 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742621 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742550 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742825 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742742 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.742825 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.742806 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.743325 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.743299 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946115c9-5d90-431d-8213-9724c9f1a430-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.744779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.744752 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.745078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.745060 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946115c9-5d90-431d-8213-9724c9f1a430-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.795413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.795345 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.795541 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.795508 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxwm\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-kube-api-access-tjxwm\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:13.840354 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:13.840329 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:14.260107 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:14.260081 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw"] Nov 02 08:27:14.261410 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:27:14.261381 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946115c9_5d90_431d_8213_9724c9f1a430.slice/crio-d3e93b6d98c9fa89eec57adb4a7a421bbbe4c93830f25bb46533985a9dc3d0b9 WatchSource:0}: Error finding container d3e93b6d98c9fa89eec57adb4a7a421bbbe4c93830f25bb46533985a9dc3d0b9: Status 404 returned error can't find the container with id d3e93b6d98c9fa89eec57adb4a7a421bbbe4c93830f25bb46533985a9dc3d0b9 Nov 02 08:27:14.263684 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:14.263650 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:27:14.263746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:14.263726 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:27:14.263795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:14.263755 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:27:15.024188 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:15.024135 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" event={"ID":"946115c9-5d90-431d-8213-9724c9f1a430","Type":"ContainerStarted","Data":"f36b75aa29b6f7340bf475ea6d332c084c39a4d40e481d089c4c3eaf52aeb39d"} Nov 02 08:27:15.024601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:15.024234 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" event={"ID":"946115c9-5d90-431d-8213-9724c9f1a430","Type":"ContainerStarted","Data":"d3e93b6d98c9fa89eec57adb4a7a421bbbe4c93830f25bb46533985a9dc3d0b9"} Nov 02 08:27:15.183976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:15.183926 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" podStartSLOduration=2.183912327 podStartE2EDuration="2.183912327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:15.179837952 +0000 UTC m=+1233.733342155" watchObservedRunningTime="2026-11-02 08:27:15.183912327 +0000 UTC m=+1233.737416523" Nov 02 08:27:15.840532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:15.840499 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:15.842215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:15.842168 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 08:27:15.842346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:15.842244 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 08:27:16.841121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:16.841081 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 08:27:16.841573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:16.841156 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 08:27:17.841131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:17.841084 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 08:27:17.841567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:17.841147 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 08:27:18.432888 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:18.432844 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:18.842143 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:18.842038 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 08:27:18.842143 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:18.842108 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 08:27:19.842071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:19.842038 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 08:27:19.842336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:19.842100 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 08:27:19.953238 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:19.953199 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:20.655074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:20.655032 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:20.840810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:20.840770 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 08:27:20.841007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:20.840856 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 08:27:21.165158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:21.165123 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:22.045049 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:22.045010 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:22.045482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:22.045451 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:22.046377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:22.046353 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:27:23.178133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:23.178097 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:24.840719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:24.840680 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:25.020016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:25.019970 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:25.188749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:25.188717 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:28.052142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.052106 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:28.231983 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.231936 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:28.383896 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.383799 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:28.588081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.588044 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg"] Nov 02 08:27:28.592486 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.592452 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:28.688636 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.688604 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cba6f9af-dockercfg-2s9tv\"" Nov 02 08:27:28.688636 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.688611 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cba6f9af\"" Nov 02 08:27:28.692864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.692843 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg"] Nov 02 08:27:28.732946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.732915 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4dzs\" (UniqueName: \"kubernetes.io/projected/2a0c8576-8ea7-408d-9a55-aacd9de4a646-kube-api-access-l4dzs\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg\" (UID: \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:28.733090 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.733076 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0c8576-8ea7-408d-9a55-aacd9de4a646-plugins-config-volume\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg\" (UID: \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:28.834007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.833978 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4dzs\" (UniqueName: \"kubernetes.io/projected/2a0c8576-8ea7-408d-9a55-aacd9de4a646-kube-api-access-l4dzs\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg\" (UID: \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:28.834217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.834099 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0c8576-8ea7-408d-9a55-aacd9de4a646-plugins-config-volume\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg\" (UID: \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:28.834836 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.834788 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0c8576-8ea7-408d-9a55-aacd9de4a646-plugins-config-volume\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg\" (UID: \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:28.894894 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.894860 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4dzs\" (UniqueName: \"kubernetes.io/projected/2a0c8576-8ea7-408d-9a55-aacd9de4a646-kube-api-access-l4dzs\") pod \"payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg\" (UID: \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\") " pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:28.917087 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:28.917060 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:29.283736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:29.283712 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg"] Nov 02 08:27:29.285824 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:27:29.285790 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0c8576_8ea7_408d_9a55_aacd9de4a646.slice/crio-15346df376fc8af5957e4a358451f3b86195eea749e3fd154fd702eba355a324 WatchSource:0}: Error finding container 15346df376fc8af5957e4a358451f3b86195eea749e3fd154fd702eba355a324: Status 404 returned error can't find the container with id 15346df376fc8af5957e4a358451f3b86195eea749e3fd154fd702eba355a324 Nov 02 08:27:30.127035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:30.126991 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" event={"ID":"2a0c8576-8ea7-408d-9a55-aacd9de4a646","Type":"ContainerStarted","Data":"d32bf91552ac51ade316641af73a8b2ec2a3febe8f107e192a901a0030d565c5"} Nov 02 08:27:30.127236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:30.127053 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" event={"ID":"2a0c8576-8ea7-408d-9a55-aacd9de4a646","Type":"ContainerStarted","Data":"15346df376fc8af5957e4a358451f3b86195eea749e3fd154fd702eba355a324"} Nov 02 08:27:30.127236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:30.127078 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:30.233324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:30.233239 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" podStartSLOduration=2.233223236 podStartE2EDuration="2.233223236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:30.233137468 +0000 UTC m=+1248.786641666" watchObservedRunningTime="2026-11-02 08:27:30.233223236 +0000 UTC m=+1248.786727432" Nov 02 08:27:33.133549 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:33.133514 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:33.844638 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:33.844604 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:36.279390 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.279354 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44"] Nov 02 08:27:36.283614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.283598 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.314830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.314798 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-dshm\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.314967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.314881 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tls-certs\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.315046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.314982 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwhm\" (UniqueName: \"kubernetes.io/projected/aef7da7f-e56f-447a-be43-eae7ac6e7da2-kube-api-access-blwhm\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.315046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.315030 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tmp-dir\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.315244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.315227 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-home\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.315418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.315393 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-model-cache\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.379829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.379783 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-108cf9\"/\"distinct-2-w3-108cf9-kserve-self-signed-certs\"" Nov 02 08:27:36.379829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.379809 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-108cf9\"/\"kube-root-ca.crt\"" Nov 02 08:27:36.379829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.379811 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-108cf9\"/\"default-dockercfg-rh5vv\"" Nov 02 08:27:36.379829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.379829 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-108cf9\"/\"openshift-service-ca.crt\"" Nov 02 08:27:36.382950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.382927 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44"] Nov 02 08:27:36.398077 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.398049 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:36.416493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.416468 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwhm\" (UniqueName: \"kubernetes.io/projected/aef7da7f-e56f-447a-be43-eae7ac6e7da2-kube-api-access-blwhm\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.416730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.416524 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tmp-dir\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.416730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.416583 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-home\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.416730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.416681 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-model-cache\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.416909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.416733 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-dshm\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.416909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.416771 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tls-certs\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.417213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.417016 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tmp-dir\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.417213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.417164 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-home\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.417391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.417336 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-model-cache\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.419798 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.419762 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-dshm\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.425986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.425938 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tls-certs\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.470064 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.470035 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwhm\" (UniqueName: \"kubernetes.io/projected/aef7da7f-e56f-447a-be43-eae7ac6e7da2-kube-api-access-blwhm\") pod \"distinct-2-w3-108cf9-kserve-756d4dd668-9gw44\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.596149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.596065 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:36.975864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:36.975837 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44"] Nov 02 08:27:36.977294 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:27:36.977257 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaef7da7f_e56f_447a_be43_eae7ac6e7da2.slice/crio-c5df461a1d703b89f3debd11b471887ae3b4c938077aee9589dde35f2d8c6ac9 WatchSource:0}: Error finding container c5df461a1d703b89f3debd11b471887ae3b4c938077aee9589dde35f2d8c6ac9: Status 404 returned error can't find the container with id c5df461a1d703b89f3debd11b471887ae3b4c938077aee9589dde35f2d8c6ac9 Nov 02 08:27:37.174072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:37.174033 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" event={"ID":"aef7da7f-e56f-447a-be43-eae7ac6e7da2","Type":"ContainerStarted","Data":"403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4"} Nov 02 08:27:37.174259 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:37.174081 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" event={"ID":"aef7da7f-e56f-447a-be43-eae7ac6e7da2","Type":"ContainerStarted","Data":"c5df461a1d703b89f3debd11b471887ae3b4c938077aee9589dde35f2d8c6ac9"} Nov 02 08:27:37.283028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:37.282921 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" podStartSLOduration=1.282904493 podStartE2EDuration="1.282904493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:37.279980403 +0000 UTC m=+1255.833484600" watchObservedRunningTime="2026-11-02 08:27:37.282904493 +0000 UTC m=+1255.836408690" Nov 02 08:27:37.596307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:37.596232 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:37.924603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:37.924567 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:40.521861 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:27:40.521824 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:27:41.137416 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:41.137389 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:27:42.852519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:42.852483 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:44.458612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.458574 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8"] Nov 02 08:27:44.464570 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.464550 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.507251 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.507213 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-dshm\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.507440 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.507328 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-model-cache\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.507440 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.507396 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tmp-dir\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.507565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.507439 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-home\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.507646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.507614 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xdl\" (UniqueName: \"kubernetes.io/projected/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-kube-api-access-c8xdl\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.507706 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.507668 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tls-certs\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.515391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.515365 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8"] Nov 02 08:27:44.557482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.557438 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"kube-root-ca.crt\"" Nov 02 08:27:44.557482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.557438 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"openshift-service-ca.crt\"" Nov 02 08:27:44.559312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.559289 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"default-dockercfg-d8n5d\"" Nov 02 08:27:44.559312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.559292 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"distinct-w2-10d2fb-kserve-self-signed-certs\"" Nov 02 08:27:44.573828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.573797 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:44.609324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.609131 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tmp-dir\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.609324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.609242 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-home\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.609324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.609309 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xdl\" (UniqueName: \"kubernetes.io/projected/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-kube-api-access-c8xdl\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.609909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.609343 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tls-certs\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.609909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.609415 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-dshm\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.609909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.609492 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-model-cache\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.611594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.610007 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-model-cache\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.611594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.610347 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tmp-dir\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.611594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.610596 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-home\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.617071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.616504 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-dshm\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.617071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.616972 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tls-certs\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.663980 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.663954 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xdl\" (UniqueName: \"kubernetes.io/projected/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-kube-api-access-c8xdl\") pod \"distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:44.777765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:44.777682 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:45.158275 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:45.158245 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8"] Nov 02 08:27:45.159405 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:27:45.159381 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf4a79e8_f9f5_4688_a58e_5c02fc2c0ca2.slice/crio-31adea840182ba4c01f34a802c3b6a706e36baccf35a4ed1762fe54e2a599472 WatchSource:0}: Error finding container 31adea840182ba4c01f34a802c3b6a706e36baccf35a4ed1762fe54e2a599472: Status 404 returned error can't find the container with id 31adea840182ba4c01f34a802c3b6a706e36baccf35a4ed1762fe54e2a599472 Nov 02 08:27:45.227447 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:45.227410 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" event={"ID":"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2","Type":"ContainerStarted","Data":"e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa"} Nov 02 08:27:45.227581 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:45.227464 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" event={"ID":"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2","Type":"ContainerStarted","Data":"31adea840182ba4c01f34a802c3b6a706e36baccf35a4ed1762fe54e2a599472"} Nov 02 08:27:45.336711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:45.336653 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" podStartSLOduration=1.336638038 podStartE2EDuration="1.336638038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:45.332851806 +0000 UTC m=+1263.886356002" watchObservedRunningTime="2026-11-02 08:27:45.336638038 +0000 UTC m=+1263.890142235" Nov 02 08:27:45.777912 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:45.777881 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:46.596364 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:46.596324 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:46.609301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:46.609272 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:47.252639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:47.252614 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:27:47.674436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:47.674398 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:48.110391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:48.110311 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:52.534106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:52.534009 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:54.778666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:54.778630 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:54.791783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:54.791757 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:55.305667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:55.305635 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:27:57.270197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.263574 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn"] Nov 02 08:27:57.271283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.271249 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.327786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.327762 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9d77b6-openshift-default-dockercfg-r279j\"" Nov 02 08:27:57.375316 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.375283 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn"] Nov 02 08:27:57.467043 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.467010 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-certs\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.467043 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.467055 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-data\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.467280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.467153 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-podinfo\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.467280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.467224 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqp62\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-kube-api-access-jqp62\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.467280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.467247 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istiod-ca-cert\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.467380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.467338 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-token\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.467414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.467384 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-credential-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.467482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.467470 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.467515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.467488 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-envoy\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.493098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.493059 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:57.568565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.568476 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.568565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.568510 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-envoy\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.568756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.568644 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-certs\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.568756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.568692 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-data\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.568756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.568736 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-podinfo\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.568898 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.568765 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqp62\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-kube-api-access-jqp62\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.568898 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.568787 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istiod-ca-cert\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.568898 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.568842 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-token\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.569124 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.568905 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-credential-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.569209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.569191 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-certs\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.569386 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.569339 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-data\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.569697 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.569636 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.569797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.569692 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-credential-socket\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.569797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.569733 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istiod-ca-cert\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.571135 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.571115 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-envoy\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.571506 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.571490 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-podinfo\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.624467 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.624441 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-token\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.624599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.624574 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqp62\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-kube-api-access-jqp62\") pod \"e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:57.886277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:57.886185 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:58.314517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:58.314488 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn"] Nov 02 08:27:58.316433 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:27:58.316404 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab5f20f_4fc2_44bd_9d08_dfce7903b6d7.slice/crio-a0a588ec97fa59bb44bcef6ba8644350217da945031cadd1b1af6266b46a9d16 WatchSource:0}: Error finding container a0a588ec97fa59bb44bcef6ba8644350217da945031cadd1b1af6266b46a9d16: Status 404 returned error can't find the container with id a0a588ec97fa59bb44bcef6ba8644350217da945031cadd1b1af6266b46a9d16 Nov 02 08:27:58.319329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:58.319296 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:27:58.319429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:58.319361 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:27:58.319429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:58.319389 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:27:59.338372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:59.338335 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" event={"ID":"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7","Type":"ContainerStarted","Data":"652b2b7569841174959b7f430d675e40ea854f5c60b705729d97ac76dfddd35f"} Nov 02 08:27:59.338778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:59.338392 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" event={"ID":"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7","Type":"ContainerStarted","Data":"a0a588ec97fa59bb44bcef6ba8644350217da945031cadd1b1af6266b46a9d16"} Nov 02 08:27:59.544089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:59.544003 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" podStartSLOduration=2.54398646 podStartE2EDuration="2.54398646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:59.495880787 +0000 UTC m=+1278.049384983" watchObservedRunningTime="2026-11-02 08:27:59.54398646 +0000 UTC m=+1278.097490659" Nov 02 08:27:59.886570 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:59.886537 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:27:59.888170 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:59.888142 6160 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 08:27:59.888337 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:27:59.888231 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 08:28:00.886973 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:00.886928 6160 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 08:28:00.887388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:00.887000 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 08:28:01.886794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:01.886748 6160 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 08:28:01.886982 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:01.886809 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 08:28:02.306079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:02.306043 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:02.532911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:02.532866 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:02.775594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:02.775561 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:02.887227 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:02.887163 6160 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 08:28:02.887401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:02.887241 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 08:28:03.886546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:03.886516 6160 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 08:28:03.887009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:03.886985 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 08:28:04.886894 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:04.886858 6160 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 08:28:04.887400 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:04.886913 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 08:28:04.985878 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:04.985837 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:05.891210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:05.891164 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:28:05.891611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:05.891573 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:28:05.892368 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:05.892350 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:28:07.461907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:07.461872 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:07.643445 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:07.643411 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:09.922995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:09.922922 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:10.573514 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:28:10.573447 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:28:12.353538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:12.353482 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:17.144158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:17.144114 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:18.335344 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:18.335302 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:18.632603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:18.632507 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:19.007102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:19.007057 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:19.403938 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:19.403846 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:19.777433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:19.777392 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:22.054168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:22.054135 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:25.859133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.859096 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd"] Nov 02 08:28:25.863312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.863293 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.912192 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.912150 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zmgsp\"" Nov 02 08:28:25.933863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.933832 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.933958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.933873 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.933958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.933898 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.934062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.934041 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.934121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.934108 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.934223 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.934206 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.934273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.934248 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.934273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.934267 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjts7\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-kube-api-access-bjts7\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.934369 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.934343 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:25.962290 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:25.962258 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd"] Nov 02 08:28:26.035559 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.035529 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.035745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.035581 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.035745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.035610 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.035745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.035627 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjts7\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-kube-api-access-bjts7\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.035745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.035658 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.035745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.035707 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.035745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.035737 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.036018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.035768 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.036018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.035841 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.036193 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.036145 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.036279 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.036259 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.036377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.036349 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.036618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.036593 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.036804 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.036776 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.038293 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.038272 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.038713 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.038693 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.090676 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.090641 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.090839 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.090707 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjts7\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-kube-api-access-bjts7\") pod \"maas-default-gateway-openshift-default-769df7846c-l54fd\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.177429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.177392 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:26.601965 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.601884 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd"] Nov 02 08:28:26.637135 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:28:26.637106 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc724ad29_9eb6_4e0c_9ff3_0f949e462de0.slice/crio-9e8db27a35e2b19acd3e8f16cb9b8d852754b2b9b288369f892b7d9cb86a927b WatchSource:0}: Error finding container 9e8db27a35e2b19acd3e8f16cb9b8d852754b2b9b288369f892b7d9cb86a927b: Status 404 returned error can't find the container with id 9e8db27a35e2b19acd3e8f16cb9b8d852754b2b9b288369f892b7d9cb86a927b Nov 02 08:28:26.639487 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.639452 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:28:26.639559 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.639532 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:28:26.639599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.639566 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:28:26.974236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:26.974195 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:27.540763 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:27.540723 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" event={"ID":"c724ad29-9eb6-4e0c-9ff3-0f949e462de0","Type":"ContainerStarted","Data":"f5dfc12cdf16ca735e2a0b2e1fc7623bdcf6ea3b577c7fd66a28132d83720e44"} Nov 02 08:28:27.540945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:27.540788 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" event={"ID":"c724ad29-9eb6-4e0c-9ff3-0f949e462de0","Type":"ContainerStarted","Data":"9e8db27a35e2b19acd3e8f16cb9b8d852754b2b9b288369f892b7d9cb86a927b"} Nov 02 08:28:27.662213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:27.662128 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podStartSLOduration=2.66211292 podStartE2EDuration="2.66211292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:27.660505306 +0000 UTC m=+1306.214009522" watchObservedRunningTime="2026-11-02 08:28:27.66211292 +0000 UTC m=+1306.215617117" Nov 02 08:28:28.177960 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:28.177927 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:28.179438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:28.179415 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l54fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:28:28.179494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:28.179468 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 08:28:29.178371 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:29.178332 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l54fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:28:29.178829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:29.178396 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 08:28:29.329201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:29.328165 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:29.986665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:29.986626 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:30.178120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:30.178082 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l54fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:28:30.178326 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:30.178148 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 08:28:30.501380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:30.501346 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:31.178710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:31.178675 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l54fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:28:31.178877 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:31.178739 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 08:28:31.996127 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:31.996090 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:32.177771 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:32.177733 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l54fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:28:32.177951 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:32.177808 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 08:28:33.178769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:33.178731 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l54fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:28:33.179209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:33.178798 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 08:28:33.829653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:33.829621 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:34.015944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:34.015902 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:34.178538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:34.178501 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l54fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 08:28:34.178760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:34.178571 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 08:28:34.203738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:34.203697 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:35.739212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:35.738520 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:35.743202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:35.740043 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:35.749276 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:35.744383 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:28:37.108682 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.108631 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:37.332354 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.332312 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:37.526762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.526714 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:37.634662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.634627 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th"] Nov 02 08:28:37.639971 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.639946 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:37.693711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.692446 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th"] Nov 02 08:28:37.724230 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.724144 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fde55a9-ba16-4854-bef3-bd7a05375218-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th\" (UID: \"0fde55a9-ba16-4854-bef3-bd7a05375218\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:37.724430 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.724272 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6f2\" (UniqueName: \"kubernetes.io/projected/0fde55a9-ba16-4854-bef3-bd7a05375218-kube-api-access-td6f2\") pod \"payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th\" (UID: \"0fde55a9-ba16-4854-bef3-bd7a05375218\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:37.734474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.734435 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f5bb4f-dockercfg-n2klz\"" Nov 02 08:28:37.734679 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.734509 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f5bb4f\"" Nov 02 08:28:37.825396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.825297 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fde55a9-ba16-4854-bef3-bd7a05375218-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th\" (UID: \"0fde55a9-ba16-4854-bef3-bd7a05375218\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:37.825396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.825387 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6f2\" (UniqueName: \"kubernetes.io/projected/0fde55a9-ba16-4854-bef3-bd7a05375218-kube-api-access-td6f2\") pod \"payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th\" (UID: \"0fde55a9-ba16-4854-bef3-bd7a05375218\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:37.826796 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.826763 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fde55a9-ba16-4854-bef3-bd7a05375218-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th\" (UID: \"0fde55a9-ba16-4854-bef3-bd7a05375218\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:37.843669 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.843636 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn"] Nov 02 08:28:37.847859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.847837 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:37.879940 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.879910 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6f2\" (UniqueName: \"kubernetes.io/projected/0fde55a9-ba16-4854-bef3-bd7a05375218-kube-api-access-td6f2\") pod \"payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th\" (UID: \"0fde55a9-ba16-4854-bef3-bd7a05375218\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:37.903306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.903265 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn"] Nov 02 08:28:37.959579 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:37.956393 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:38.028191 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.028142 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec334a13-fe07-4868-b0b8-e70625372832-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn\" (UID: \"ec334a13-fe07-4868-b0b8-e70625372832\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:38.028425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.028396 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtt2p\" (UniqueName: \"kubernetes.io/projected/ec334a13-fe07-4868-b0b8-e70625372832-kube-api-access-xtt2p\") pod \"payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn\" (UID: \"ec334a13-fe07-4868-b0b8-e70625372832\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:38.129395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.129308 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtt2p\" (UniqueName: \"kubernetes.io/projected/ec334a13-fe07-4868-b0b8-e70625372832-kube-api-access-xtt2p\") pod \"payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn\" (UID: \"ec334a13-fe07-4868-b0b8-e70625372832\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:38.129770 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.129418 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec334a13-fe07-4868-b0b8-e70625372832-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn\" (UID: \"ec334a13-fe07-4868-b0b8-e70625372832\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:38.130099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.130079 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec334a13-fe07-4868-b0b8-e70625372832-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn\" (UID: \"ec334a13-fe07-4868-b0b8-e70625372832\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:38.183699 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.183670 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtt2p\" (UniqueName: \"kubernetes.io/projected/ec334a13-fe07-4868-b0b8-e70625372832-kube-api-access-xtt2p\") pod \"payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn\" (UID: \"ec334a13-fe07-4868-b0b8-e70625372832\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:38.339017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.338990 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th"] Nov 02 08:28:38.340393 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:28:38.340364 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fde55a9_ba16_4854_bef3_bd7a05375218.slice/crio-5cbfd354354cd97db2076bd0b978de8c4d98cc855c5b025b894281bab416f245 WatchSource:0}: Error finding container 5cbfd354354cd97db2076bd0b978de8c4d98cc855c5b025b894281bab416f245: Status 404 returned error can't find the container with id 5cbfd354354cd97db2076bd0b978de8c4d98cc855c5b025b894281bab416f245 Nov 02 08:28:38.463290 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.463259 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:38.623330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.623299 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" event={"ID":"0fde55a9-ba16-4854-bef3-bd7a05375218","Type":"ContainerStarted","Data":"6e56bc8178bc0614585bb44dbf87184bdff6b2c7acb2b1a63e867ea017b025a0"} Nov 02 08:28:38.623513 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.623347 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" event={"ID":"0fde55a9-ba16-4854-bef3-bd7a05375218","Type":"ContainerStarted","Data":"5cbfd354354cd97db2076bd0b978de8c4d98cc855c5b025b894281bab416f245"} Nov 02 08:28:38.623513 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.623415 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:38.736014 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.735955 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" podStartSLOduration=1.7359369629999999 podStartE2EDuration="1.735936963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:38.731297961 +0000 UTC m=+1317.284802157" watchObservedRunningTime="2026-11-02 08:28:38.735936963 +0000 UTC m=+1317.289441160" Nov 02 08:28:38.832865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:38.832837 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn"] Nov 02 08:28:38.833554 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:28:38.833524 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec334a13_fe07_4868_b0b8_e70625372832.slice/crio-aa74cb8f7f74f535c6bc948f3e77b73e3f723ed0e10e32fbb40c0ece184de199 WatchSource:0}: Error finding container aa74cb8f7f74f535c6bc948f3e77b73e3f723ed0e10e32fbb40c0ece184de199: Status 404 returned error can't find the container with id aa74cb8f7f74f535c6bc948f3e77b73e3f723ed0e10e32fbb40c0ece184de199 Nov 02 08:28:39.495144 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.495107 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77"] Nov 02 08:28:39.499425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.499407 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.547871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.547839 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-108cf9-openshift-default-dockercfg-gprcr\"" Nov 02 08:28:39.598799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.598763 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77"] Nov 02 08:28:39.633381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.633344 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" event={"ID":"ec334a13-fe07-4868-b0b8-e70625372832","Type":"ContainerStarted","Data":"1bfa96e25e52d30967f0493d98c87b8daa9908459fe0440df8487c036e82d08c"} Nov 02 08:28:39.633582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.633398 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" event={"ID":"ec334a13-fe07-4868-b0b8-e70625372832","Type":"ContainerStarted","Data":"aa74cb8f7f74f535c6bc948f3e77b73e3f723ed0e10e32fbb40c0ece184de199"} Nov 02 08:28:39.633582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.633427 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:39.648320 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.648283 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-envoy\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.648494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.648341 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-credential-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.648494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.648417 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-token\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.648494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.648453 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cca66b-c26c-48f6-acaa-6c44537a259e-istiod-ca-cert\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.648655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.648543 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-podinfo\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.648655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.648569 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-certs\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.648655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.648592 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsmpq\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-kube-api-access-vsmpq\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.648805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.648711 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-data\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.648805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.648771 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.742046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.741996 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" podStartSLOduration=2.741981252 podStartE2EDuration="2.741981252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:39.740800552 +0000 UTC m=+1318.294304762" watchObservedRunningTime="2026-11-02 08:28:39.741981252 +0000 UTC m=+1318.295485497" Nov 02 08:28:39.750351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750282 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-data\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750385 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750499 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-envoy\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750541 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-credential-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750648 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750594 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-token\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750645 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cca66b-c26c-48f6-acaa-6c44537a259e-istiod-ca-cert\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750671 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-data\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750822 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750734 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750820 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-podinfo\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750852 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-certs\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750896 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsmpq\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-kube-api-access-vsmpq\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.750994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.750918 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-credential-socket\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.751140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.751124 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-certs\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.751563 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.751543 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cca66b-c26c-48f6-acaa-6c44537a259e-istiod-ca-cert\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.753149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.753121 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-envoy\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.753401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.753381 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-podinfo\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.805767 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.805737 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-token\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.806009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.805992 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsmpq\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-kube-api-access-vsmpq\") pod \"e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:39.813862 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:39.813841 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:40.238989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:40.238958 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77"] Nov 02 08:28:40.239392 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:28:40.239366 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cca66b_c26c_48f6_acaa_6c44537a259e.slice/crio-f176971f6189a228539d62fe465b605116ad3bd494a856007575d21bf5d56e30 WatchSource:0}: Error finding container f176971f6189a228539d62fe465b605116ad3bd494a856007575d21bf5d56e30: Status 404 returned error can't find the container with id f176971f6189a228539d62fe465b605116ad3bd494a856007575d21bf5d56e30 Nov 02 08:28:40.242023 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:40.241979 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:28:40.242121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:40.242079 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:28:40.242165 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:40.242122 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:28:40.523588 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:28:40.523495 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:28:40.641967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:40.641933 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" event={"ID":"f1cca66b-c26c-48f6-acaa-6c44537a259e","Type":"ContainerStarted","Data":"e5fc1c7f57085fe6e75e6f5ed62b7b1c76e04ef956286497306fde2d90445095"} Nov 02 08:28:40.642202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:40.641990 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" event={"ID":"f1cca66b-c26c-48f6-acaa-6c44537a259e","Type":"ContainerStarted","Data":"f176971f6189a228539d62fe465b605116ad3bd494a856007575d21bf5d56e30"} Nov 02 08:28:40.756202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:40.755695 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" podStartSLOduration=1.755672674 podStartE2EDuration="1.755672674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:40.752877132 +0000 UTC m=+1319.306381328" watchObservedRunningTime="2026-11-02 08:28:40.755672674 +0000 UTC m=+1319.309176873" Nov 02 08:28:40.818273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:40.814059 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:41.332682 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:41.332641 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:41.831328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:41.831287 6160 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:28:41.831746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:41.831357 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 08:28:42.360795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:42.360765 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:42.831205 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:42.822276 6160 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:28:42.831205 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:42.822354 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 08:28:43.831321 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:43.831275 6160 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:28:43.831507 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:43.831341 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 08:28:44.058228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:44.058153 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:44.256960 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:44.256922 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:44.823200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:44.822533 6160 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:28:44.823200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:44.822599 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 08:28:45.827599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:45.827563 6160 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:28:45.828161 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:45.828126 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 08:28:45.944038 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:45.944005 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:46.814942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:46.814905 6160 patch_prober.go:29] interesting pod/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 08:28:46.815133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:46.814965 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 08:28:47.280314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.280271 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:47.506233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.506189 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:47.680575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.680538 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9"] Nov 02 08:28:47.685838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.685815 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:28:47.735223 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.735168 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b26f03\"" Nov 02 08:28:47.758947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.758914 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ht5\" (UniqueName: \"kubernetes.io/projected/004c0899-355d-45d5-8cff-743aba69ff08-kube-api-access-f9ht5\") pod \"payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9\" (UID: \"004c0899-355d-45d5-8cff-743aba69ff08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:28:47.758947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.758952 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004c0899-355d-45d5-8cff-743aba69ff08-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9\" (UID: \"004c0899-355d-45d5-8cff-743aba69ff08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:28:47.781346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.781314 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b26f03-dockercfg-mdr4l\"" Nov 02 08:28:47.786682 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.786651 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9"] Nov 02 08:28:47.834953 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.834924 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:47.836387 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.836223 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:47.846029 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.844988 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:28:47.855082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.855040 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4"] Nov 02 08:28:47.862591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.860402 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ht5\" (UniqueName: \"kubernetes.io/projected/004c0899-355d-45d5-8cff-743aba69ff08-kube-api-access-f9ht5\") pod \"payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9\" (UID: \"004c0899-355d-45d5-8cff-743aba69ff08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:28:47.862591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.860467 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004c0899-355d-45d5-8cff-743aba69ff08-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9\" (UID: \"004c0899-355d-45d5-8cff-743aba69ff08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:28:47.862591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.862529 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004c0899-355d-45d5-8cff-743aba69ff08-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9\" (UID: \"004c0899-355d-45d5-8cff-743aba69ff08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:28:47.863189 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.863156 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:47.884338 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.882894 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4"] Nov 02 08:28:47.894513 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.894428 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:47.916162 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.916122 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ht5\" (UniqueName: \"kubernetes.io/projected/004c0899-355d-45d5-8cff-743aba69ff08-kube-api-access-f9ht5\") pod \"payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9\" (UID: \"004c0899-355d-45d5-8cff-743aba69ff08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:28:47.962297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.962206 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037a02af-4aa5-4abe-9d4b-3659f87869b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4\" (UID: \"037a02af-4aa5-4abe-9d4b-3659f87869b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:47.962469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:47.962348 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6rk\" (UniqueName: \"kubernetes.io/projected/037a02af-4aa5-4abe-9d4b-3659f87869b0-kube-api-access-wg6rk\") pod \"payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4\" (UID: \"037a02af-4aa5-4abe-9d4b-3659f87869b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:48.001024 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.000986 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:28:48.063418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.063287 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037a02af-4aa5-4abe-9d4b-3659f87869b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4\" (UID: \"037a02af-4aa5-4abe-9d4b-3659f87869b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:48.063418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.063361 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6rk\" (UniqueName: \"kubernetes.io/projected/037a02af-4aa5-4abe-9d4b-3659f87869b0-kube-api-access-wg6rk\") pod \"payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4\" (UID: \"037a02af-4aa5-4abe-9d4b-3659f87869b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:48.064291 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.064263 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037a02af-4aa5-4abe-9d4b-3659f87869b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4\" (UID: \"037a02af-4aa5-4abe-9d4b-3659f87869b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:48.121591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.120725 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6rk\" (UniqueName: \"kubernetes.io/projected/037a02af-4aa5-4abe-9d4b-3659f87869b0-kube-api-access-wg6rk\") pod \"payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4\" (UID: \"037a02af-4aa5-4abe-9d4b-3659f87869b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:48.188165 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.188130 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:48.398577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.398548 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9"] Nov 02 08:28:48.399282 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:28:48.399254 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004c0899_355d_45d5_8cff_743aba69ff08.slice/crio-96af192a544d4ddc6c3683bbcaa483bd39ae1a2c1e0d3ffa8480f13c8e95be93 WatchSource:0}: Error finding container 96af192a544d4ddc6c3683bbcaa483bd39ae1a2c1e0d3ffa8480f13c8e95be93: Status 404 returned error can't find the container with id 96af192a544d4ddc6c3683bbcaa483bd39ae1a2c1e0d3ffa8480f13c8e95be93 Nov 02 08:28:48.558855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.558830 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4"] Nov 02 08:28:48.560114 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:28:48.560089 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037a02af_4aa5_4abe_9d4b_3659f87869b0.slice/crio-d6c7272ab57b0ffaf5c2ecb4aca8d7541614c33d85fce76a9e6f6fe827ba3626 WatchSource:0}: Error finding container d6c7272ab57b0ffaf5c2ecb4aca8d7541614c33d85fce76a9e6f6fe827ba3626: Status 404 returned error can't find the container with id d6c7272ab57b0ffaf5c2ecb4aca8d7541614c33d85fce76a9e6f6fe827ba3626 Nov 02 08:28:48.700905 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.700867 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" event={"ID":"037a02af-4aa5-4abe-9d4b-3659f87869b0","Type":"ContainerStarted","Data":"c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10"} Nov 02 08:28:48.701079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.700922 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" event={"ID":"037a02af-4aa5-4abe-9d4b-3659f87869b0","Type":"ContainerStarted","Data":"d6c7272ab57b0ffaf5c2ecb4aca8d7541614c33d85fce76a9e6f6fe827ba3626"} Nov 02 08:28:48.701079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.700959 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:48.702664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.702642 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" event={"ID":"004c0899-355d-45d5-8cff-743aba69ff08","Type":"ContainerStarted","Data":"3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32"} Nov 02 08:28:48.709746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.702877 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" event={"ID":"004c0899-355d-45d5-8cff-743aba69ff08","Type":"ContainerStarted","Data":"96af192a544d4ddc6c3683bbcaa483bd39ae1a2c1e0d3ffa8480f13c8e95be93"} Nov 02 08:28:48.709746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.703511 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:28:48.807393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.807284 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" podStartSLOduration=1.807270156 podStartE2EDuration="1.807270156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:48.805425918 +0000 UTC m=+1327.358930118" watchObservedRunningTime="2026-11-02 08:28:48.807270156 +0000 UTC m=+1327.360774772" Nov 02 08:28:48.913906 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:48.913850 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" podStartSLOduration=1.9138358709999999 podStartE2EDuration="1.913835871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:48.911022274 +0000 UTC m=+1327.464526471" watchObservedRunningTime="2026-11-02 08:28:48.913835871 +0000 UTC m=+1327.467340067" Nov 02 08:28:49.637309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:49.637283 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:28:50.645931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:50.645900 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:28:53.230802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.230765 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:53.882614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.882580 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt"] Nov 02 08:28:53.886860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.886839 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:53.946493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.946460 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-dshm\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:53.946673 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.946582 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmv64\" (UniqueName: \"kubernetes.io/projected/a0176aba-5186-4993-84f9-7e5c8b79d1a9-kube-api-access-fmv64\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:53.946673 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.946611 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tls-certs\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:53.946797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.946747 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-home\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:53.946855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.946794 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-model-cache\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:53.946855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.946843 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tmp-dir\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:53.983129 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.983102 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-10d2fb\"/\"distinct-2-w2-10d2fb-kserve-self-signed-certs\"" Nov 02 08:28:53.987428 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:53.987397 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt"] Nov 02 08:28:54.019864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.019833 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:54.048439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.048412 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-dshm\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.048572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.048489 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmv64\" (UniqueName: \"kubernetes.io/projected/a0176aba-5186-4993-84f9-7e5c8b79d1a9-kube-api-access-fmv64\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.048572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.048514 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tls-certs\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.048572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.048553 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-home\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.048706 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.048577 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-model-cache\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.048706 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.048612 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tmp-dir\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.049028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.048999 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-home\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.049122 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.049012 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-model-cache\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.049122 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.049055 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tmp-dir\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.050863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.050843 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-dshm\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.051206 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.051162 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tls-certs\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.102004 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.101972 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmv64\" (UniqueName: \"kubernetes.io/projected/a0176aba-5186-4993-84f9-7e5c8b79d1a9-kube-api-access-fmv64\") pod \"distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.201623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.201581 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:54.585605 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.585572 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt"] Nov 02 08:28:54.586544 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:28:54.586516 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0176aba_5186_4993_84f9_7e5c8b79d1a9.slice/crio-e0a9db2a37d7fe8bf26c86d2b21050dfe0a045fa55b4bf02a8d6700d34c54411 WatchSource:0}: Error finding container e0a9db2a37d7fe8bf26c86d2b21050dfe0a045fa55b4bf02a8d6700d34c54411: Status 404 returned error can't find the container with id e0a9db2a37d7fe8bf26c86d2b21050dfe0a045fa55b4bf02a8d6700d34c54411 Nov 02 08:28:54.745064 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.745030 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" event={"ID":"a0176aba-5186-4993-84f9-7e5c8b79d1a9","Type":"ContainerStarted","Data":"cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39"} Nov 02 08:28:54.745064 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.745071 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" event={"ID":"a0176aba-5186-4993-84f9-7e5c8b79d1a9","Type":"ContainerStarted","Data":"e0a9db2a37d7fe8bf26c86d2b21050dfe0a045fa55b4bf02a8d6700d34c54411"} Nov 02 08:28:54.852756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:54.852663 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" podStartSLOduration=1.852649178 podStartE2EDuration="1.852649178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:54.852496607 +0000 UTC m=+1333.406000803" watchObservedRunningTime="2026-11-02 08:28:54.852649178 +0000 UTC m=+1333.406153375" Nov 02 08:28:55.201817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:55.201787 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:28:55.864221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:55.864146 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:58.647859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:58.647774 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:59.316833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.316798 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv"] Nov 02 08:28:59.321095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.321078 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.414034 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.414003 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702d810a-cd06-4807-9470-17613fda7434-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.414214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.414080 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qr66\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-kube-api-access-2qr66\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.414281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.414245 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.414328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.414305 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.414395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.414377 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702d810a-cd06-4807-9470-17613fda7434-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.414433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.414421 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.414480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.414466 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.414601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.414586 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.414648 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.414611 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.516580 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.516539 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qr66\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-kube-api-access-2qr66\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.516750 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.516628 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.516750 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.516678 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.516750 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.516728 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702d810a-cd06-4807-9470-17613fda7434-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.516971 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.516775 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.516971 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.516837 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.516971 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.516902 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.516971 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.516931 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.517121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.516990 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702d810a-cd06-4807-9470-17613fda7434-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.517304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.517275 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-data\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.517988 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.517964 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702d810a-cd06-4807-9470-17613fda7434-istiod-ca-cert\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.518478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.518457 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-credential-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.519653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.519093 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-certs\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.519976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.519940 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-socket\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.521279 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.521255 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv"] Nov 02 08:28:59.522104 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.522061 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702d810a-cd06-4807-9470-17613fda7434-istio-podinfo\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.522282 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.522262 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-envoy\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.571219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.571127 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-istio-token\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.571641 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.571618 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qr66\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-kube-api-access-2qr66\") pod \"e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.635210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.635145 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:28:59.723488 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.723460 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:28:59.724621 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:28:59.724598 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:29:00.098378 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:00.098347 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv"] Nov 02 08:29:00.098859 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:29:00.098835 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702d810a_cd06_4807_9470_17613fda7434.slice/crio-8c1397097083e2ecd853dba96be0d93eb351e2d01b7b249693188a69233a3f1f WatchSource:0}: Error finding container 8c1397097083e2ecd853dba96be0d93eb351e2d01b7b249693188a69233a3f1f: Status 404 returned error can't find the container with id 8c1397097083e2ecd853dba96be0d93eb351e2d01b7b249693188a69233a3f1f Nov 02 08:29:00.101546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:00.101514 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:29:00.101600 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:00.101581 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:29:00.101631 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:00.101611 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:29:00.788750 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:00.788713 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" event={"ID":"702d810a-cd06-4807-9470-17613fda7434","Type":"ContainerStarted","Data":"1c674d95b45afff467f63b17d99aa6546039817a1e5363d375c37d007bc63bb6"} Nov 02 08:29:00.789307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:00.788776 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" event={"ID":"702d810a-cd06-4807-9470-17613fda7434","Type":"ContainerStarted","Data":"8c1397097083e2ecd853dba96be0d93eb351e2d01b7b249693188a69233a3f1f"} Nov 02 08:29:00.948132 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:00.948069 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" podStartSLOduration=1.948055277 podStartE2EDuration="1.948055277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:29:00.945241794 +0000 UTC m=+1339.498746000" watchObservedRunningTime="2026-11-02 08:29:00.948055277 +0000 UTC m=+1339.501559470" Nov 02 08:29:01.635395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:01.635354 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:29:01.637305 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:01.637277 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 08:29:01.637455 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:01.637331 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 08:29:02.635819 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:02.635785 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 08:29:02.636251 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:02.635841 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 08:29:03.532870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:03.532830 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:03.635702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:03.635666 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 08:29:03.635876 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:03.635717 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 08:29:04.202557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:04.202523 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:29:04.239256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:04.239223 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:29:04.636727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:04.636622 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 08:29:04.636727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:04.636697 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 08:29:04.847899 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:04.847869 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:29:05.355044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:05.354992 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:05.635940 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:05.635841 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 08:29:05.635940 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:05.635896 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 08:29:06.636081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:06.636038 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 08:29:06.636565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:06.636095 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 08:29:07.640675 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:07.640645 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:29:07.641089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:07.641070 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:29:07.641727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:07.641710 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:29:08.356329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:08.356297 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:10.522592 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:29:10.522550 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:29:13.125802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:13.125765 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:17.671909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.671875 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb"] Nov 02 08:29:17.677160 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.677136 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.775497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.775462 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f5bb4f-serving-cert\"" Nov 02 08:29:17.779414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.779389 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb"] Nov 02 08:29:17.848583 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.848556 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52882ba1-9a64-446b-bf7b-5324e1d1a545-openshift-service-ca\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.848752 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.848613 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-maas-api-tls\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.848752 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.848725 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-metrics-certs\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.848838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.848819 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7877\" (UniqueName: \"kubernetes.io/projected/52882ba1-9a64-446b-bf7b-5324e1d1a545-kube-api-access-m7877\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.888764 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.888734 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:17.949925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.949811 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-metrics-certs\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.949925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.949895 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7877\" (UniqueName: \"kubernetes.io/projected/52882ba1-9a64-446b-bf7b-5324e1d1a545-kube-api-access-m7877\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.950169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.949967 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52882ba1-9a64-446b-bf7b-5324e1d1a545-openshift-service-ca\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.950169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.950031 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-maas-api-tls\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.950805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.950778 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52882ba1-9a64-446b-bf7b-5324e1d1a545-openshift-service-ca\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.952579 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.952560 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-metrics-certs\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:17.952687 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:17.952665 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-maas-api-tls\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:18.004115 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:18.004081 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7877\" (UniqueName: \"kubernetes.io/projected/52882ba1-9a64-446b-bf7b-5324e1d1a545-kube-api-access-m7877\") pod \"maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:18.107802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:18.107760 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:18.291388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:18.291302 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:18.665984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:18.665944 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb"] Nov 02 08:29:18.666862 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:29:18.666827 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52882ba1_9a64_446b_bf7b_5324e1d1a545.slice/crio-61e03e7e6b4b458474d12600136a5bb6bf165987ac60c388b73abefd3865901c WatchSource:0}: Error finding container 61e03e7e6b4b458474d12600136a5bb6bf165987ac60c388b73abefd3865901c: Status 404 returned error can't find the container with id 61e03e7e6b4b458474d12600136a5bb6bf165987ac60c388b73abefd3865901c Nov 02 08:29:18.668449 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:18.668426 6160 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:29:18.931098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:18.931013 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" event={"ID":"52882ba1-9a64-446b-bf7b-5324e1d1a545","Type":"ContainerStarted","Data":"61e03e7e6b4b458474d12600136a5bb6bf165987ac60c388b73abefd3865901c"} Nov 02 08:29:19.939490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:19.939455 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" event={"ID":"52882ba1-9a64-446b-bf7b-5324e1d1a545","Type":"ContainerStarted","Data":"5f8edae10915dbbd3cb6ab16d7c1fd7d8d88f8038a330f0b0f07c3dd8b740610"} Nov 02 08:29:19.939899 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:19.939572 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:20.052989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:20.052937 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" podStartSLOduration=2.164791993 podStartE2EDuration="3.052922561s" totalImagesPullingTime="888.130568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:29:20.050067778 +0000 UTC m=+1358.603571976" watchObservedRunningTime="2026-11-02 08:29:20.052922561 +0000 UTC m=+1358.606426758" Nov 02 08:29:20.274295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:20.273994 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:23.035480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:23.035433 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:25.977587 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:25.977539 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:29:27.874792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:27.874753 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:32.689525 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:32.689489 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:32.910400 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:32.910365 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:34.532972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:34.532936 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:37.091287 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.091246 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd"] Nov 02 08:29:37.096246 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.096221 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.162124 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.162087 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.162317 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.162169 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.162317 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.162226 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.162317 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.162278 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rb99\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-kube-api-access-9rb99\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.162317 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.162313 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.162482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.162340 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6917d45-23e7-49f5-b505-3536e900c6f8-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.162482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.162412 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.162544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.162489 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.162544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.162522 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.263875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.263840 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.263875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.263878 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264100 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.263894 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rb99\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-kube-api-access-9rb99\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264100 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264016 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264100 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264058 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6917d45-23e7-49f5-b505-3536e900c6f8-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264124 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264286 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264299 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264333 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264393 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-certs\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264535 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264483 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264535 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264490 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-data\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264804 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264781 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6917d45-23e7-49f5-b505-3536e900c6f8-istiod-ca-cert\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.264922 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.264800 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-credential-socket\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.266689 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.266667 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-podinfo\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.266794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.266749 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-envoy\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.287140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.287112 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd"] Nov 02 08:29:37.317980 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.317940 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rb99\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-kube-api-access-9rb99\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.318207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.318150 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-token\") pod \"e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.409820 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.409722 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:37.758728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.758696 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:37.834603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.834568 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd"] Nov 02 08:29:37.835924 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:29:37.835895 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6917d45_23e7_49f5_b505_3536e900c6f8.slice/crio-0019f3d40a4038e97ae5195ad27f8acc1bf32f069935a7c3c93841113d769191 WatchSource:0}: Error finding container 0019f3d40a4038e97ae5195ad27f8acc1bf32f069935a7c3c93841113d769191: Status 404 returned error can't find the container with id 0019f3d40a4038e97ae5195ad27f8acc1bf32f069935a7c3c93841113d769191 Nov 02 08:29:37.838716 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.838680 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:29:37.838850 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.838768 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:29:37.838850 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:37.838814 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:29:38.094082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:38.093971 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" event={"ID":"c6917d45-23e7-49f5-b505-3536e900c6f8","Type":"ContainerStarted","Data":"5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73"} Nov 02 08:29:38.094082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:38.094037 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" event={"ID":"c6917d45-23e7-49f5-b505-3536e900c6f8","Type":"ContainerStarted","Data":"0019f3d40a4038e97ae5195ad27f8acc1bf32f069935a7c3c93841113d769191"} Nov 02 08:29:38.207497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:38.207442 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" podStartSLOduration=1.2074270089999999 podStartE2EDuration="1.207427009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:29:38.205939107 +0000 UTC m=+1376.759443349" watchObservedRunningTime="2026-11-02 08:29:38.207427009 +0000 UTC m=+1376.760931206" Nov 02 08:29:38.419261 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:38.412610 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:38.426199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:38.422328 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 08:29:38.426199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:38.422411 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 08:29:39.425202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:39.419653 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 08:29:39.425202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:39.419718 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 08:29:40.417200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:40.411568 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 08:29:40.417200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:40.411628 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 08:29:40.545677 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:29:40.545636 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:29:41.420203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:41.414622 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 08:29:41.420203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:41.414690 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 08:29:42.418203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:42.410524 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 08:29:42.418203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:42.410584 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 08:29:42.895231 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:42.895124 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:43.422672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:43.422635 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 08:29:43.423359 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:43.423327 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 08:29:44.410963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:44.410926 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 08:29:44.411148 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:44.410981 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 08:29:45.415233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:45.415202 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:45.415716 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:45.415491 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:45.416197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:45.416166 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:29:45.756062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:45.755972 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:47.922632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:47.922595 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:52.647084 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:52.647045 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:57.564090 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:29:57.564050 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:02.350426 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.350380 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:02.627921 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.627835 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb"] Nov 02 08:30:02.632107 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.632088 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.714485 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.714445 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0cf78f1-hcnwb\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.714485 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.714496 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0cf78f1-hcnwb\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.714770 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.714740 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp2fw\" (UniqueName: \"kubernetes.io/projected/49c62352-c357-4856-8f73-4183fbba0c89-kube-api-access-xp2fw\") pod \"maas-api-revoke-sub-b0cf78f1-hcnwb\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.728264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.728234 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:30:02.728264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.728238 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:30:02.732872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.732847 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb"] Nov 02 08:30:02.815873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.815837 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp2fw\" (UniqueName: \"kubernetes.io/projected/49c62352-c357-4856-8f73-4183fbba0c89-kube-api-access-xp2fw\") pod \"maas-api-revoke-sub-b0cf78f1-hcnwb\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.816048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.815983 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0cf78f1-hcnwb\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.816048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.816009 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0cf78f1-hcnwb\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.816791 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.816773 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0cf78f1-hcnwb\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.818507 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.818486 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0cf78f1-hcnwb\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.869049 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.869019 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp2fw\" (UniqueName: \"kubernetes.io/projected/49c62352-c357-4856-8f73-4183fbba0c89-kube-api-access-xp2fw\") pod \"maas-api-revoke-sub-b0cf78f1-hcnwb\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:02.946015 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:02.945977 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:03.318939 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:03.318911 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb"] Nov 02 08:30:03.319868 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:30:03.319842 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c62352_c357_4856_8f73_4183fbba0c89.slice/crio-57825e3c7da84ad25888e91ddda41b06321a9e65f3e28403ace582fb50f290a5 WatchSource:0}: Error finding container 57825e3c7da84ad25888e91ddda41b06321a9e65f3e28403ace582fb50f290a5: Status 404 returned error can't find the container with id 57825e3c7da84ad25888e91ddda41b06321a9e65f3e28403ace582fb50f290a5 Nov 02 08:30:04.316495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:04.316456 6160 generic.go:356] "Generic (PLEG): container finished" podID="49c62352-c357-4856-8f73-4183fbba0c89" containerID="07df5ada0c8076b67a26a3ecf1398650c2f18e9caa38e12967eb9655e424a303" exitCode=0 Nov 02 08:30:04.316906 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:04.316541 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" event={"ID":"49c62352-c357-4856-8f73-4183fbba0c89","Type":"ContainerDied","Data":"07df5ada0c8076b67a26a3ecf1398650c2f18e9caa38e12967eb9655e424a303"} Nov 02 08:30:04.316906 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:04.316596 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" event={"ID":"49c62352-c357-4856-8f73-4183fbba0c89","Type":"ContainerStarted","Data":"57825e3c7da84ad25888e91ddda41b06321a9e65f3e28403ace582fb50f290a5"} Nov 02 08:30:05.470332 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.470308 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:05.547110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.547071 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-xks-ca\") pod \"49c62352-c357-4856-8f73-4183fbba0c89\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " Nov 02 08:30:05.547303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.547159 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-openshift-ca\") pod \"49c62352-c357-4856-8f73-4183fbba0c89\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " Nov 02 08:30:05.547303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.547259 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c62352-c357-4856-8f73-4183fbba0c89-kube-api-access-xp2fw\" (UniqueName: \"kubernetes.io/projected/49c62352-c357-4856-8f73-4183fbba0c89-kube-api-access-xp2fw\") pod \"49c62352-c357-4856-8f73-4183fbba0c89\" (UID: \"49c62352-c357-4856-8f73-4183fbba0c89\") " Nov 02 08:30:05.547579 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.547551 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-openshift-ca" pod "49c62352-c357-4856-8f73-4183fbba0c89" (UID: "49c62352-c357-4856-8f73-4183fbba0c89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:05.547754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.547739 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-openshift-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:05.549485 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.549452 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c62352-c357-4856-8f73-4183fbba0c89-kube-api-access-xp2fw" pod "49c62352-c357-4856-8f73-4183fbba0c89" (UID: "49c62352-c357-4856-8f73-4183fbba0c89"). InnerVolumeSpecName "kube-api-access-xp2fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:05.549485 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.549469 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-xks-ca" pod "49c62352-c357-4856-8f73-4183fbba0c89" (UID: "49c62352-c357-4856-8f73-4183fbba0c89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:30:05.649480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.649436 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp2fw\" (UniqueName: \"kubernetes.io/projected/49c62352-c357-4856-8f73-4183fbba0c89-kube-api-access-xp2fw\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:05.649480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.649469 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c62352-c357-4856-8f73-4183fbba0c89-maas-api-xks-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:30:05.719860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:05.719828 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:06.334942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.334917 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" Nov 02 08:30:06.334942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.334914 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb" event={"ID":"49c62352-c357-4856-8f73-4183fbba0c89","Type":"ContainerDied","Data":"57825e3c7da84ad25888e91ddda41b06321a9e65f3e28403ace582fb50f290a5"} Nov 02 08:30:06.335147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.334971 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57825e3c7da84ad25888e91ddda41b06321a9e65f3e28403ace582fb50f290a5" Nov 02 08:30:06.865949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.865912 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr"] Nov 02 08:30:06.866577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.866561 6160 cpu_manager.go:414] "containerMap: removing container" podUID="49c62352-c357-4856-8f73-4183fbba0c89" containerName="revoke-keys" Nov 02 08:30:06.866641 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.866579 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c62352-c357-4856-8f73-4183fbba0c89" containerName="revoke-keys" Nov 02 08:30:06.866696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.866687 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c62352-c357-4856-8f73-4183fbba0c89" containerName="revoke-keys" Nov 02 08:30:06.870346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.870331 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.964816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.964781 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.964816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.964816 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.965102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.964944 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.965102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.965003 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.965102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.965066 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.965329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.965140 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.965329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.965222 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.965329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.965271 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.965329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.965301 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbl2\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-kube-api-access-tsbl2\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:06.971322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:06.971298 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr"] Nov 02 08:30:07.066783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.066749 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.066958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.066792 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbl2\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-kube-api-access-tsbl2\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.066958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.066834 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.066958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.066852 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.066958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.066898 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.066958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.066931 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.067270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.066974 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.067270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.067040 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.067270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.067069 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.067431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.067366 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.067431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.067410 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-data\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.067667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.067645 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-certs\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.067751 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.067729 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-credential-socket\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.067818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.067801 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istiod-ca-cert\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.069798 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.069777 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-envoy\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.069986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.069965 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-podinfo\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.120823 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.120750 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-token\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.120967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.120939 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbl2\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-kube-api-access-tsbl2\") pod \"e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.185363 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.185329 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:07.342769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.342735 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:07.624564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.624535 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr"] Nov 02 08:30:07.625334 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:30:07.625300 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342ae15a_1d5c_4860_9eaa_0fc9b2cb27c0.slice/crio-cfd0119a8ed3ee12100265e9f123276a1551187a079413666bba85a740a5e0b5 WatchSource:0}: Error finding container cfd0119a8ed3ee12100265e9f123276a1551187a079413666bba85a740a5e0b5: Status 404 returned error can't find the container with id cfd0119a8ed3ee12100265e9f123276a1551187a079413666bba85a740a5e0b5 Nov 02 08:30:07.627902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.627868 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:30:07.627961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.627935 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:30:07.628000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:07.627962 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:30:08.351241 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:08.351205 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" event={"ID":"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0","Type":"ContainerStarted","Data":"ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94"} Nov 02 08:30:08.351649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:08.351267 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" event={"ID":"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0","Type":"ContainerStarted","Data":"cfd0119a8ed3ee12100265e9f123276a1551187a079413666bba85a740a5e0b5"} Nov 02 08:30:08.531201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:08.530002 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" podStartSLOduration=2.5299818099999998 podStartE2EDuration="2.52998181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:08.528689498 +0000 UTC m=+1407.082193729" watchObservedRunningTime="2026-11-02 08:30:08.52998181 +0000 UTC m=+1407.083486008" Nov 02 08:30:09.185528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:09.185495 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:09.186898 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:09.186873 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 08:30:09.186999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:09.186922 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 08:30:10.186101 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:10.186062 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 08:30:10.186523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:10.186114 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 08:30:10.431367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:10.430461 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:10.545141 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:30:10.545048 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:30:11.095242 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:11.095202 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:11.186257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:11.186220 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 08:30:11.186731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:11.186277 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 08:30:11.656221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:11.656156 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:12.185756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:12.185723 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 08:30:12.185973 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:12.185779 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 08:30:12.254001 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:12.253967 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:13.132311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:13.132266 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:13.186523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:13.186487 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 08:30:13.186700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:13.186543 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 08:30:13.903563 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:13.903519 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:14.186038 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:14.186002 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 08:30:14.186346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:14.186312 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 08:30:14.586712 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:14.586607 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:15.190833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:15.190802 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:15.191234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:15.191211 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:15.191841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:15.191825 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:30:16.484919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:16.484872 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:17.056661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:17.056614 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:17.423759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:17.423720 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:17.807410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:17.807323 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:19.065062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:19.065017 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:19.287864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:19.287826 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:19.454469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:19.454432 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:20.826547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:20.826478 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:21.513658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:21.513622 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:21.903692 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:21.903597 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:22.038939 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.038888 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d"] Nov 02 08:30:22.046976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.046945 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.092976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.092940 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-envoy\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.093159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.093035 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e452-05c5-4715-801d-ffa370eadee1-istiod-ca-cert\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.093159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.093096 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-socket\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.093159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.093139 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-credential-socket\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.093408 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.093282 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-istio-token\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.093408 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.093338 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vnn\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-kube-api-access-n4vnn\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.093529 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.093469 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-data\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.093529 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.093510 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-certs\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.093638 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.093537 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e452-05c5-4715-801d-ffa370eadee1-istio-podinfo\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.102200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.102132 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d"] Nov 02 08:30:22.113318 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.113269 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:22.143558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.143529 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a25202-openshift-default-dockercfg-66mhr\"" Nov 02 08:30:22.194608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.194578 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e452-05c5-4715-801d-ffa370eadee1-istiod-ca-cert\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.194810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.194623 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-socket\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.194810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.194653 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-credential-socket\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.194810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.194697 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-istio-token\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.194810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.194745 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vnn\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-kube-api-access-n4vnn\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.195046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.194806 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-data\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.195046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.194865 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-certs\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.195046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.194897 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e452-05c5-4715-801d-ffa370eadee1-istio-podinfo\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.195046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.194934 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-envoy\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.195222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.195068 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-socket\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.195222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.195100 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-credential-socket\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.195562 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.195536 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-data\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.195851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.195544 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e452-05c5-4715-801d-ffa370eadee1-istiod-ca-cert\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.195851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.195607 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-certs\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.197408 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.197374 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-envoy\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.197747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.197725 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e452-05c5-4715-801d-ffa370eadee1-istio-podinfo\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.265571 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.265523 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-istio-token\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.287578 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.287531 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vnn\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-kube-api-access-n4vnn\") pod \"e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.325199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.322283 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:22.362274 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.362245 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:22.498247 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.498129 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:22.790249 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:30:22.790220 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e44e452_05c5_4715_801d_ffa370eadee1.slice/crio-527d86217def818d0d77ac3239dfb32fe78cc2e88c41b3bf0af8fc7b50b2e5d7 WatchSource:0}: Error finding container 527d86217def818d0d77ac3239dfb32fe78cc2e88c41b3bf0af8fc7b50b2e5d7: Status 404 returned error can't find the container with id 527d86217def818d0d77ac3239dfb32fe78cc2e88c41b3bf0af8fc7b50b2e5d7 Nov 02 08:30:22.791688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.791662 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d"] Nov 02 08:30:22.792987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.792949 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:30:22.793068 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.793014 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:30:22.793068 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:22.793042 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:30:23.136278 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.136164 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:23.299007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.298972 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:23.469568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.469535 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" event={"ID":"3e44e452-05c5-4715-801d-ffa370eadee1","Type":"ContainerStarted","Data":"516cb709cdb3f8e98b3fa1c6e417f6e26d68f56e86810e28dc5f16ec137f7332"} Nov 02 08:30:23.469765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.469597 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" event={"ID":"3e44e452-05c5-4715-801d-ffa370eadee1","Type":"ContainerStarted","Data":"527d86217def818d0d77ac3239dfb32fe78cc2e88c41b3bf0af8fc7b50b2e5d7"} Nov 02 08:30:23.592136 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.592043 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" podStartSLOduration=2.592025345 podStartE2EDuration="2.592025345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:23.590542324 +0000 UTC m=+1422.144046523" watchObservedRunningTime="2026-11-02 08:30:23.592025345 +0000 UTC m=+1422.145529543" Nov 02 08:30:23.655086 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.655050 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf"] Nov 02 08:30:23.662030 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.662000 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:23.712057 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.712022 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-00cc0e\"" Nov 02 08:30:23.712280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.712221 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-00cc0e-dockercfg-d4p5r\"" Nov 02 08:30:23.712394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.712367 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf"] Nov 02 08:30:23.732295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.732208 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606d0f-465b-487e-85ea-d6a7f45256ba-plugins-config-volume\") pod \"payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf\" (UID: \"52606d0f-465b-487e-85ea-d6a7f45256ba\") " pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:23.732431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.732296 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2q9g\" (UniqueName: \"kubernetes.io/projected/52606d0f-465b-487e-85ea-d6a7f45256ba-kube-api-access-w2q9g\") pod \"payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf\" (UID: \"52606d0f-465b-487e-85ea-d6a7f45256ba\") " pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:23.833938 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.833902 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2q9g\" (UniqueName: \"kubernetes.io/projected/52606d0f-465b-487e-85ea-d6a7f45256ba-kube-api-access-w2q9g\") pod \"payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf\" (UID: \"52606d0f-465b-487e-85ea-d6a7f45256ba\") " pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:23.834155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.834133 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606d0f-465b-487e-85ea-d6a7f45256ba-plugins-config-volume\") pod \"payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf\" (UID: \"52606d0f-465b-487e-85ea-d6a7f45256ba\") " pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:23.834889 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.834860 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606d0f-465b-487e-85ea-d6a7f45256ba-plugins-config-volume\") pod \"payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf\" (UID: \"52606d0f-465b-487e-85ea-d6a7f45256ba\") " pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:23.886940 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:23.886907 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2q9g\" (UniqueName: \"kubernetes.io/projected/52606d0f-465b-487e-85ea-d6a7f45256ba-kube-api-access-w2q9g\") pod \"payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf\" (UID: \"52606d0f-465b-487e-85ea-d6a7f45256ba\") " pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:24.001141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.001056 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:24.362655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.362570 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:24.368276 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.368250 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:24.375897 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.375872 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf"] Nov 02 08:30:24.376457 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:30:24.376427 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52606d0f_465b_487e_85ea_d6a7f45256ba.slice/crio-e7d66c3dd719f761019670cbf5775e86cb5606352490ef26e2cc9e3eb8c16c19 WatchSource:0}: Error finding container e7d66c3dd719f761019670cbf5775e86cb5606352490ef26e2cc9e3eb8c16c19: Status 404 returned error can't find the container with id e7d66c3dd719f761019670cbf5775e86cb5606352490ef26e2cc9e3eb8c16c19 Nov 02 08:30:24.480240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.480195 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" event={"ID":"52606d0f-465b-487e-85ea-d6a7f45256ba","Type":"ContainerStarted","Data":"0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0"} Nov 02 08:30:24.480422 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.480263 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" event={"ID":"52606d0f-465b-487e-85ea-d6a7f45256ba","Type":"ContainerStarted","Data":"e7d66c3dd719f761019670cbf5775e86cb5606352490ef26e2cc9e3eb8c16c19"} Nov 02 08:30:24.480422 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.480293 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:24.480819 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.480800 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:24.481753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.481732 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:30:24.590747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:24.590697 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" podStartSLOduration=1.5906818870000001 podStartE2EDuration="1.590681887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:24.588424492 +0000 UTC m=+1423.141928710" watchObservedRunningTime="2026-11-02 08:30:24.590681887 +0000 UTC m=+1423.144186083" Nov 02 08:30:26.555669 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:26.555627 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:26.778326 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:26.778290 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:26.946057 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:26.946022 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:28.166154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:28.166122 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:28.358991 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:28.358949 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:28.869594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:28.869557 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:30.274078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:30.274041 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:30.525350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:30.525255 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:30.692934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:30.692896 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:30.858755 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:30.858661 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:31.026086 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.026049 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:31.176773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.176740 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52"] Nov 02 08:30:31.180888 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.180867 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:31.236075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.236039 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52"] Nov 02 08:30:31.276301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.276263 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a25202-dockercfg-xmqkl\"" Nov 02 08:30:31.276709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.276311 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a25202\"" Nov 02 08:30:31.346835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.344285 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb"] Nov 02 08:30:31.363985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.355754 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fbfc5e6-88b9-4576-b121-0e4ef0780368-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52\" (UID: \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:31.363985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.355932 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xfj\" (UniqueName: \"kubernetes.io/projected/1fbfc5e6-88b9-4576-b121-0e4ef0780368-kube-api-access-29xfj\") pod \"payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52\" (UID: \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:31.397195 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.384478 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:31.424517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.407445 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb"] Nov 02 08:30:31.456990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.456954 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xfj\" (UniqueName: \"kubernetes.io/projected/1fbfc5e6-88b9-4576-b121-0e4ef0780368-kube-api-access-29xfj\") pod \"payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52\" (UID: \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:31.457245 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.457222 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fbfc5e6-88b9-4576-b121-0e4ef0780368-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52\" (UID: \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:31.458083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.458060 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fbfc5e6-88b9-4576-b121-0e4ef0780368-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52\" (UID: \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:31.510800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.510773 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xfj\" (UniqueName: \"kubernetes.io/projected/1fbfc5e6-88b9-4576-b121-0e4ef0780368-kube-api-access-29xfj\") pod \"payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52\" (UID: \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:31.558731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.558698 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb\" (UID: \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\") " pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:31.558943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.558830 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbzp\" (UniqueName: \"kubernetes.io/projected/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-kube-api-access-mgbzp\") pod \"payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb\" (UID: \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\") " pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:31.660161 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.660127 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb\" (UID: \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\") " pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:31.660356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.660272 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbzp\" (UniqueName: \"kubernetes.io/projected/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-kube-api-access-mgbzp\") pod \"payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb\" (UID: \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\") " pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:31.660851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.660831 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb\" (UID: \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\") " pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:31.713598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.713516 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbzp\" (UniqueName: \"kubernetes.io/projected/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-kube-api-access-mgbzp\") pod \"payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb\" (UID: \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\") " pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:31.750632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.750600 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:31.796143 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:31.796113 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:32.128069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.128044 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb"] Nov 02 08:30:32.128841 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:30:32.128810 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3f86dd_fa7e_459a_a49d_c2f710f1bead.slice/crio-cc0f754df3a40f58d96e44674206c976676ce1b2b446a016691a40c600e02cc4 WatchSource:0}: Error finding container cc0f754df3a40f58d96e44674206c976676ce1b2b446a016691a40c600e02cc4: Status 404 returned error can't find the container with id cc0f754df3a40f58d96e44674206c976676ce1b2b446a016691a40c600e02cc4 Nov 02 08:30:32.233523 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:30:32.233492 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbfc5e6_88b9_4576_b121_0e4ef0780368.slice/crio-dd97a4334e2faf0115119b632aad088204705e3e88c2b8661f1dd05001c588b9 WatchSource:0}: Error finding container dd97a4334e2faf0115119b632aad088204705e3e88c2b8661f1dd05001c588b9: Status 404 returned error can't find the container with id dd97a4334e2faf0115119b632aad088204705e3e88c2b8661f1dd05001c588b9 Nov 02 08:30:32.235156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.235125 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52"] Nov 02 08:30:32.551673 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.551635 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" event={"ID":"dd3f86dd-fa7e-459a-a49d-c2f710f1bead","Type":"ContainerStarted","Data":"72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa"} Nov 02 08:30:32.552139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.551734 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" event={"ID":"dd3f86dd-fa7e-459a-a49d-c2f710f1bead","Type":"ContainerStarted","Data":"cc0f754df3a40f58d96e44674206c976676ce1b2b446a016691a40c600e02cc4"} Nov 02 08:30:32.552139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.551884 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:32.553471 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.553437 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" event={"ID":"1fbfc5e6-88b9-4576-b121-0e4ef0780368","Type":"ContainerStarted","Data":"1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c"} Nov 02 08:30:32.553599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.553490 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:32.553599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.553506 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" event={"ID":"1fbfc5e6-88b9-4576-b121-0e4ef0780368","Type":"ContainerStarted","Data":"dd97a4334e2faf0115119b632aad088204705e3e88c2b8661f1dd05001c588b9"} Nov 02 08:30:32.660813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.660762 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" podStartSLOduration=1.6607480940000001 podStartE2EDuration="1.660748094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:32.657783512 +0000 UTC m=+1431.211287709" watchObservedRunningTime="2026-11-02 08:30:32.660748094 +0000 UTC m=+1431.214252291" Nov 02 08:30:32.769272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:32.769215 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" podStartSLOduration=1.769198694 podStartE2EDuration="1.769198694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:32.764738928 +0000 UTC m=+1431.318243125" watchObservedRunningTime="2026-11-02 08:30:32.769198694 +0000 UTC m=+1431.322702885" Nov 02 08:30:34.328606 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:34.328570 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:35.499146 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:35.499112 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:30:36.066919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:36.066878 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:36.790124 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:36.790089 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:37.768112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:37.768077 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:37.975776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:37.975727 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:39.904999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:39.904968 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:40.523009 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:30:40.522973 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:30:40.693226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:40.693158 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:40.857818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:40.857718 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:41.262203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.259550 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn"] Nov 02 08:30:41.266528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.266501 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:41.295038 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.294999 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn"] Nov 02 08:30:41.296847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.296810 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc93aac4-9fbd-4739-a09c-be525981d22b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn\" (UID: \"dc93aac4-9fbd-4739-a09c-be525981d22b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:41.301825 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.301712 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzwr\" (UniqueName: \"kubernetes.io/projected/dc93aac4-9fbd-4739-a09c-be525981d22b-kube-api-access-6jzwr\") pod \"payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn\" (UID: \"dc93aac4-9fbd-4739-a09c-be525981d22b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:41.315776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.315745 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-19ee94\"" Nov 02 08:30:41.333528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.333373 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-19ee94-dockercfg-4cwx4\"" Nov 02 08:30:41.403373 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.403340 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc93aac4-9fbd-4739-a09c-be525981d22b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn\" (UID: \"dc93aac4-9fbd-4739-a09c-be525981d22b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:41.403512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.403467 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzwr\" (UniqueName: \"kubernetes.io/projected/dc93aac4-9fbd-4739-a09c-be525981d22b-kube-api-access-6jzwr\") pod \"payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn\" (UID: \"dc93aac4-9fbd-4739-a09c-be525981d22b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:41.404024 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.404004 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc93aac4-9fbd-4739-a09c-be525981d22b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn\" (UID: \"dc93aac4-9fbd-4739-a09c-be525981d22b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:41.457098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.457068 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzwr\" (UniqueName: \"kubernetes.io/projected/dc93aac4-9fbd-4739-a09c-be525981d22b-kube-api-access-6jzwr\") pod \"payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn\" (UID: \"dc93aac4-9fbd-4739-a09c-be525981d22b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:41.633917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.633823 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-19ee94-dockercfg-4cwx4\"" Nov 02 08:30:41.635939 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:41.635921 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:42.013833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:42.013801 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn"] Nov 02 08:30:42.014401 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:30:42.014372 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc93aac4_9fbd_4739_a09c_be525981d22b.slice/crio-e6c3c1fa1beb84e5fd85b8b927c447202d50a9bcc8d16f4379786d7653dafc0f WatchSource:0}: Error finding container e6c3c1fa1beb84e5fd85b8b927c447202d50a9bcc8d16f4379786d7653dafc0f: Status 404 returned error can't find the container with id e6c3c1fa1beb84e5fd85b8b927c447202d50a9bcc8d16f4379786d7653dafc0f Nov 02 08:30:42.649686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:42.649633 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" event={"ID":"dc93aac4-9fbd-4739-a09c-be525981d22b","Type":"ContainerStarted","Data":"219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71"} Nov 02 08:30:42.650125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:42.649737 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" event={"ID":"dc93aac4-9fbd-4739-a09c-be525981d22b","Type":"ContainerStarted","Data":"e6c3c1fa1beb84e5fd85b8b927c447202d50a9bcc8d16f4379786d7653dafc0f"} Nov 02 08:30:42.650125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:42.649788 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:42.758083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:42.758017 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" podStartSLOduration=1.758000791 podStartE2EDuration="1.758000791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:42.756048775 +0000 UTC m=+1441.309553001" watchObservedRunningTime="2026-11-02 08:30:42.758000791 +0000 UTC m=+1441.311504989" Nov 02 08:30:43.563217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:43.563164 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:30:43.563392 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:43.563365 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:30:44.868638 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:44.868598 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:46.039447 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:46.039408 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:50.250783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:50.250696 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:52.209664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:52.209621 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:52.419714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:52.419669 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:52.589845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:52.589750 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:53.661105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:53.661074 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:30:55.589797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:55.589763 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:55.806903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:55.806858 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:55.984334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:55.984196 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:56.404934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.404854 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn"] Nov 02 08:30:56.409700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.409679 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:30:56.416852 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.416827 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee121a2-369e-435c-ac8a-bee177b9a246-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn\" (UID: \"0ee121a2-369e-435c-ac8a-bee177b9a246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:30:56.416971 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.416902 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsf6d\" (UniqueName: \"kubernetes.io/projected/0ee121a2-369e-435c-ac8a-bee177b9a246-kube-api-access-fsf6d\") pod \"payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn\" (UID: \"0ee121a2-369e-435c-ac8a-bee177b9a246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:30:56.463727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.463692 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn"] Nov 02 08:30:56.504726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.504695 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c7f7d7\"" Nov 02 08:30:56.504726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.504709 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c7f7d7-dockercfg-bcd2c\"" Nov 02 08:30:56.517752 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.517722 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee121a2-369e-435c-ac8a-bee177b9a246-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn\" (UID: \"0ee121a2-369e-435c-ac8a-bee177b9a246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:30:56.517919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.517800 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsf6d\" (UniqueName: \"kubernetes.io/projected/0ee121a2-369e-435c-ac8a-bee177b9a246-kube-api-access-fsf6d\") pod \"payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn\" (UID: \"0ee121a2-369e-435c-ac8a-bee177b9a246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:30:56.518601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.518578 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee121a2-369e-435c-ac8a-bee177b9a246-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn\" (UID: \"0ee121a2-369e-435c-ac8a-bee177b9a246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:30:56.571986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.571950 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsf6d\" (UniqueName: \"kubernetes.io/projected/0ee121a2-369e-435c-ac8a-bee177b9a246-kube-api-access-fsf6d\") pod \"payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn\" (UID: \"0ee121a2-369e-435c-ac8a-bee177b9a246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:30:56.724341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:56.724308 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:30:57.100585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:57.100558 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn"] Nov 02 08:30:57.101214 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:30:57.101159 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee121a2_369e_435c_ac8a_bee177b9a246.slice/crio-0f4c74b1f0fa1b8eb51484bc82285d44684219337a714d82848edfadedbe6367 WatchSource:0}: Error finding container 0f4c74b1f0fa1b8eb51484bc82285d44684219337a714d82848edfadedbe6367: Status 404 returned error can't find the container with id 0f4c74b1f0fa1b8eb51484bc82285d44684219337a714d82848edfadedbe6367 Nov 02 08:30:57.772340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:57.772296 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" event={"ID":"0ee121a2-369e-435c-ac8a-bee177b9a246","Type":"ContainerStarted","Data":"ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837"} Nov 02 08:30:57.772793 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:57.772370 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" event={"ID":"0ee121a2-369e-435c-ac8a-bee177b9a246","Type":"ContainerStarted","Data":"0f4c74b1f0fa1b8eb51484bc82285d44684219337a714d82848edfadedbe6367"} Nov 02 08:30:57.772793 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:57.772630 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:30:57.883657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:30:57.883590 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" podStartSLOduration=1.883576008 podStartE2EDuration="1.883576008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:57.879688919 +0000 UTC m=+1456.433193135" watchObservedRunningTime="2026-11-02 08:30:57.883576008 +0000 UTC m=+1456.437080204" Nov 02 08:31:00.618569 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:00.618535 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:01.562016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:01.561978 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:03.026407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:03.026365 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:03.821255 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:03.821213 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:05.434648 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:05.434609 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:08.783689 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:08.783659 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:31:10.222417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:10.222376 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:10.523062 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:31:10.522969 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:31:14.732917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.732884 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg"] Nov 02 08:31:14.738887 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.738864 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.835660 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.835622 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-00cc0e-serving-cert\"" Nov 02 08:31:14.839784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.839753 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg"] Nov 02 08:31:14.853107 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.853077 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-openshift-service-ca\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.853277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.853134 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-metrics-certs\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.853330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.853273 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.853486 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.853468 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfl2\" (UniqueName: \"kubernetes.io/projected/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-kube-api-access-fgfl2\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.942218 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.942164 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:14.954569 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.954547 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfl2\" (UniqueName: \"kubernetes.io/projected/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-kube-api-access-fgfl2\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.954670 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.954631 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-openshift-service-ca\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.954670 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.954664 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-metrics-certs\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.954739 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.954691 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.954816 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:31:14.954803 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-serving-cert: secret "maas-api-e2e-api-a-00cc0e-serving-cert" not found Nov 02 08:31:14.954876 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:31:14.954867 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls podName:43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a nodeName:}" failed. No retries permitted until 2026-11-02 08:31:15.454850279 +0000 UTC m=+1474.008354468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls") pod "maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" (UID: "43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a") : secret "maas-api-e2e-api-a-00cc0e-serving-cert" not found Nov 02 08:31:14.955320 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.955291 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-openshift-service-ca\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:14.957272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:14.957255 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-metrics-certs\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:15.009909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:15.009805 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfl2\" (UniqueName: \"kubernetes.io/projected/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-kube-api-access-fgfl2\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:15.095776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:15.095740 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:15.273156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:15.273067 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:15.461009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:15.460970 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:15.463658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:15.463634 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls\") pod \"maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:15.652843 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:15.652755 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:16.036829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:16.036793 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg"] Nov 02 08:31:16.042445 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:31:16.042413 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c8a1b4_cf9e_45c0_a092_c64a2ece9c4a.slice/crio-1cbc0463af3e7a89ec4798ea2aabb5b80e133d0701e7bc35c4e61075410d5a88 WatchSource:0}: Error finding container 1cbc0463af3e7a89ec4798ea2aabb5b80e133d0701e7bc35c4e61075410d5a88: Status 404 returned error can't find the container with id 1cbc0463af3e7a89ec4798ea2aabb5b80e133d0701e7bc35c4e61075410d5a88 Nov 02 08:31:16.916719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:16.916680 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" event={"ID":"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a","Type":"ContainerStarted","Data":"1cbc0463af3e7a89ec4798ea2aabb5b80e133d0701e7bc35c4e61075410d5a88"} Nov 02 08:31:17.925944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:17.925908 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" event={"ID":"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a","Type":"ContainerStarted","Data":"4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477"} Nov 02 08:31:17.926482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:17.926141 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:18.038249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:18.038161 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" podStartSLOduration=3.019583904 podStartE2EDuration="4.038140824s" totalImagesPullingTime="1.01855692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:18.037695662 +0000 UTC m=+1476.591199862" watchObservedRunningTime="2026-11-02 08:31:18.038140824 +0000 UTC m=+1476.591645022" Nov 02 08:31:19.917837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:19.917798 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:23.939394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:23.939362 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:31:24.403957 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.403875 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf"] Nov 02 08:31:24.414761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.414725 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.464198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.464137 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bca68b-serving-cert\"" Nov 02 08:31:24.465932 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.465904 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf"] Nov 02 08:31:24.508668 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.508628 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.508833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.508781 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-metrics-certs\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.508833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.508825 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dq7b\" (UniqueName: \"kubernetes.io/projected/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-kube-api-access-8dq7b\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.508924 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.508909 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.609882 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.609848 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.610073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.609971 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-metrics-certs\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.610073 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:31:24.610009 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-serving-cert: secret "maas-api-e2e-ipp-a-bca68b-serving-cert" not found Nov 02 08:31:24.610073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.610019 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dq7b\" (UniqueName: \"kubernetes.io/projected/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-kube-api-access-8dq7b\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.610243 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:31:24.610091 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls podName:bec2d528-ffaa-48a0-91a0-15b7a85c7e44 nodeName:}" failed. No retries permitted until 2026-11-02 08:31:25.110068947 +0000 UTC m=+1483.663573142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls") pod "maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" (UID: "bec2d528-ffaa-48a0-91a0-15b7a85c7e44") : secret "maas-api-e2e-ipp-a-bca68b-serving-cert" not found Nov 02 08:31:24.610243 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.610124 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.610811 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.610790 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.612629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.612611 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-metrics-certs\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.618984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.618958 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:24.663660 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.663635 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dq7b\" (UniqueName: \"kubernetes.io/projected/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-kube-api-access-8dq7b\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:24.770795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.770752 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:24.916702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:24.916609 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:25.116587 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:25.116552 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:25.119341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:25.119317 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:25.195578 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:25.195477 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:25.328781 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:25.328742 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:25.701443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:25.701418 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf"] Nov 02 08:31:25.702722 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:31:25.702692 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec2d528_ffaa_48a0_91a0_15b7a85c7e44.slice/crio-9e8685dff4f9b41d8a7cbcad55ef16d926755f33bebee9853335cc49629136aa WatchSource:0}: Error finding container 9e8685dff4f9b41d8a7cbcad55ef16d926755f33bebee9853335cc49629136aa: Status 404 returned error can't find the container with id 9e8685dff4f9b41d8a7cbcad55ef16d926755f33bebee9853335cc49629136aa Nov 02 08:31:25.990379 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:25.990295 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" event={"ID":"bec2d528-ffaa-48a0-91a0-15b7a85c7e44","Type":"ContainerStarted","Data":"9e8685dff4f9b41d8a7cbcad55ef16d926755f33bebee9853335cc49629136aa"} Nov 02 08:31:26.999541 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:26.999500 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" event={"ID":"bec2d528-ffaa-48a0-91a0-15b7a85c7e44","Type":"ContainerStarted","Data":"d8679e1b3d90c2a32dce855902b41b2e4fc39460ddcacd971b97d7c3bf41de62"} Nov 02 08:31:27.000004 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:26.999596 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:27.107749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:27.107693 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" podStartSLOduration=2.185343371 podStartE2EDuration="3.107675816s" totalImagesPullingTime="922.332445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:27.105972104 +0000 UTC m=+1485.659476304" watchObservedRunningTime="2026-11-02 08:31:27.107675816 +0000 UTC m=+1485.661180014" Nov 02 08:31:27.795839 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:27.795806 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:29.630990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:29.630945 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:31.089332 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:31.089299 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:33.012563 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:33.012527 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:31:34.380759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:34.380720 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:34.572384 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:34.572323 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:34.736705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:34.736660 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:35.833440 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:35.833399 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:36.290965 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:36.290928 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:39.566987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:39.566949 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:39.828388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:39.828283 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:39.999223 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:39.999152 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:40.522347 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:31:40.522310 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:31:40.683038 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:40.683000 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:40.933460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:40.933409 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:41.126902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:41.126863 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:41.807918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:41.807883 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:31:41.838239 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:41.838207 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:31:45.219428 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:45.219389 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:45.476625 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:45.476496 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:45.672253 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:45.672220 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:45.900594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:45.900491 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc"] Nov 02 08:31:45.916805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:45.915496 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:31:45.949630 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:45.949593 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc"] Nov 02 08:31:45.986728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:45.986697 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-69a9bb\"" Nov 02 08:31:45.987397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:45.987373 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-69a9bb-dockercfg-sqc9d\"" Nov 02 08:31:46.003532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:46.003508 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702a68bf-93ae-43bb-92af-8061cc7ced66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc\" (UID: \"702a68bf-93ae-43bb-92af-8061cc7ced66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:31:46.003677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:46.003594 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45m9g\" (UniqueName: \"kubernetes.io/projected/702a68bf-93ae-43bb-92af-8061cc7ced66-kube-api-access-45m9g\") pod \"payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc\" (UID: \"702a68bf-93ae-43bb-92af-8061cc7ced66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:31:46.105204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:46.105143 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45m9g\" (UniqueName: \"kubernetes.io/projected/702a68bf-93ae-43bb-92af-8061cc7ced66-kube-api-access-45m9g\") pod \"payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc\" (UID: \"702a68bf-93ae-43bb-92af-8061cc7ced66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:31:46.105383 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:46.105366 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702a68bf-93ae-43bb-92af-8061cc7ced66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc\" (UID: \"702a68bf-93ae-43bb-92af-8061cc7ced66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:31:46.106070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:46.106052 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702a68bf-93ae-43bb-92af-8061cc7ced66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc\" (UID: \"702a68bf-93ae-43bb-92af-8061cc7ced66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:31:46.160505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:46.160433 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45m9g\" (UniqueName: \"kubernetes.io/projected/702a68bf-93ae-43bb-92af-8061cc7ced66-kube-api-access-45m9g\") pod \"payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc\" (UID: \"702a68bf-93ae-43bb-92af-8061cc7ced66\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:31:46.266018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:46.265983 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:31:46.640495 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:31:46.640470 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702a68bf_93ae_43bb_92af_8061cc7ced66.slice/crio-7fd58eab8c99139c24ad189879c104b1abc6425138e93f43deb8914599ea79f4 WatchSource:0}: Error finding container 7fd58eab8c99139c24ad189879c104b1abc6425138e93f43deb8914599ea79f4: Status 404 returned error can't find the container with id 7fd58eab8c99139c24ad189879c104b1abc6425138e93f43deb8914599ea79f4 Nov 02 08:31:46.648470 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:46.641751 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc"] Nov 02 08:31:47.146154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:47.146123 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" event={"ID":"702a68bf-93ae-43bb-92af-8061cc7ced66","Type":"ContainerStarted","Data":"4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0"} Nov 02 08:31:47.146370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:47.146187 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" event={"ID":"702a68bf-93ae-43bb-92af-8061cc7ced66","Type":"ContainerStarted","Data":"7fd58eab8c99139c24ad189879c104b1abc6425138e93f43deb8914599ea79f4"} Nov 02 08:31:47.146370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:47.146233 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:31:47.279868 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:47.279799 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" podStartSLOduration=2.279779151 podStartE2EDuration="2.279779151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:47.277539267 +0000 UTC m=+1505.831043464" watchObservedRunningTime="2026-11-02 08:31:47.279779151 +0000 UTC m=+1505.833283348" Nov 02 08:31:50.741304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:50.741251 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:55.767807 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:55.767770 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:58.157149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:31:58.157120 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:32:00.733101 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:00.733047 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:02.961859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:02.961825 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:03.243990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:03.243893 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:05.232604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:05.232564 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:05.521872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:05.521783 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:05.800759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:05.800645 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:10.430807 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.430769 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb"] Nov 02 08:32:10.435510 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.435494 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.485601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.485567 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c7f7d7-serving-cert\"" Nov 02 08:32:10.530752 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:32:10.530713 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:32:10.535356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.535333 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb"] Nov 02 08:32:10.546753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.546726 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlm69\" (UniqueName: \"kubernetes.io/projected/71436aad-0a0c-4d55-9f18-3c147206e3b0-kube-api-access-dlm69\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.546933 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.546882 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-metrics-certs\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.547011 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.546982 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.547054 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.547038 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71436aad-0a0c-4d55-9f18-3c147206e3b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.648724 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.648686 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-metrics-certs\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.648916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.648801 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.648916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.648829 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71436aad-0a0c-4d55-9f18-3c147206e3b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.648916 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:32:10.648889 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-serving-cert: secret "maas-api-e2e-api-b-c7f7d7-serving-cert" not found Nov 02 08:32:10.649085 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.648939 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlm69\" (UniqueName: \"kubernetes.io/projected/71436aad-0a0c-4d55-9f18-3c147206e3b0-kube-api-access-dlm69\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.649085 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:32:10.648965 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls podName:71436aad-0a0c-4d55-9f18-3c147206e3b0 nodeName:}" failed. No retries permitted until 2026-11-02 08:32:11.148946308 +0000 UTC m=+1529.702450483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls") pod "maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" (UID: "71436aad-0a0c-4d55-9f18-3c147206e3b0") : secret "maas-api-e2e-api-b-c7f7d7-serving-cert" not found Nov 02 08:32:10.649472 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.649441 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71436aad-0a0c-4d55-9f18-3c147206e3b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.651407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.651390 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-metrics-certs\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.709288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.709212 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlm69\" (UniqueName: \"kubernetes.io/projected/71436aad-0a0c-4d55-9f18-3c147206e3b0-kube-api-access-dlm69\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:10.728345 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:10.728268 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:11.032490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:11.032401 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:11.156658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:11.156623 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:11.159227 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:11.159200 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls\") pod \"maas-api-e2e-api-b-c7f7d7-f86766578-bwndb\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:11.273702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:11.273663 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:11.349538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:11.349459 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:11.731097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:11.731057 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb"] Nov 02 08:32:11.731563 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:32:11.731440 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71436aad_0a0c_4d55_9f18_3c147206e3b0.slice/crio-ac760468a0048f1ce0ea84e1f7af6cb3a144507a1b028821ab185934b5629263 WatchSource:0}: Error finding container ac760468a0048f1ce0ea84e1f7af6cb3a144507a1b028821ab185934b5629263: Status 404 returned error can't find the container with id ac760468a0048f1ce0ea84e1f7af6cb3a144507a1b028821ab185934b5629263 Nov 02 08:32:12.343594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:12.343555 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" event={"ID":"71436aad-0a0c-4d55-9f18-3c147206e3b0","Type":"ContainerStarted","Data":"ac760468a0048f1ce0ea84e1f7af6cb3a144507a1b028821ab185934b5629263"} Nov 02 08:32:13.353526 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:13.353487 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" event={"ID":"71436aad-0a0c-4d55-9f18-3c147206e3b0","Type":"ContainerStarted","Data":"17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8"} Nov 02 08:32:13.354023 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:13.353727 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:13.470344 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:13.470288 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" podStartSLOduration=2.654076966 podStartE2EDuration="3.470273434s" totalImagesPullingTime="816.196468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:32:13.466811627 +0000 UTC m=+1532.020315847" watchObservedRunningTime="2026-11-02 08:32:13.470273434 +0000 UTC m=+1532.023777630" Nov 02 08:32:16.032613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:16.032571 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:19.367647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:19.367618 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:32:21.158460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:21.158377 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:21.476810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:21.476758 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:26.611813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:26.611775 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:30.544982 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:30.544941 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg"] Nov 02 08:32:31.084994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:31.084952 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:31.360539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:31.360453 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:31.612565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:31.612465 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:32.085592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:32.085557 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:35.571204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:35.553381 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" podUID="2a0c8576-8ea7-408d-9a55-aacd9de4a646" containerName="payload-processing" containerID="cri-o://d32bf91552ac51ade316641af73a8b2ec2a3febe8f107e192a901a0030d565c5" gracePeriod=25 Nov 02 08:32:38.002153 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:38.002106 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:40.542007 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:32:40.541964 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:32:41.135226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:41.135161 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" podUID="2a0c8576-8ea7-408d-9a55-aacd9de4a646" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:32:42.533469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:42.533357 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:43.248754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:43.248707 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:43.736056 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:43.736015 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:45.650845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:45.650801 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:46.711493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:46.711455 6160 generic.go:356] "Generic (PLEG): container finished" podID="3e44e452-05c5-4715-801d-ffa370eadee1" containerID="516cb709cdb3f8e98b3fa1c6e417f6e26d68f56e86810e28dc5f16ec137f7332" exitCode=0 Nov 02 08:32:46.711922 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:46.711527 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" event={"ID":"3e44e452-05c5-4715-801d-ffa370eadee1","Type":"ContainerDied","Data":"516cb709cdb3f8e98b3fa1c6e417f6e26d68f56e86810e28dc5f16ec137f7332"} Nov 02 08:32:46.711962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:46.711942 6160 scope.go:122] "RemoveContainer" containerID="516cb709cdb3f8e98b3fa1c6e417f6e26d68f56e86810e28dc5f16ec137f7332" Nov 02 08:32:46.713577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:46.713549 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:32:46.713642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:46.713625 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:32:46.713676 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:46.713660 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:32:47.282565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:47.282530 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:47.748797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:47.748751 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" event={"ID":"3e44e452-05c5-4715-801d-ffa370eadee1","Type":"ContainerStarted","Data":"c00e39643974a83d43bfaafff0d25eb620af65f32c612d175e173ec4ddc09b23"} Nov 02 08:32:48.133103 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:48.133001 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:48.363399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:48.363361 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:32:48.363704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:48.363680 6160 patch_prober.go:29] interesting pod/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:32:48.363764 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:48.363744 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 08:32:48.537389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:48.537350 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:48.943325 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:48.943290 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:49.292443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:49.292361 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:49.362627 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:49.362590 6160 patch_prober.go:29] interesting pod/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:32:49.362833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:49.362658 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 08:32:50.363035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:50.362994 6160 patch_prober.go:29] interesting pod/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:32:50.363508 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:50.363067 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 08:32:51.135656 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:51.135605 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" podUID="2a0c8576-8ea7-408d-9a55-aacd9de4a646" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:32:51.363656 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:51.363236 6160 patch_prober.go:29] interesting pod/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:32:51.363656 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:51.363308 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 08:32:52.363573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:52.363533 6160 patch_prober.go:29] interesting pod/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 08:32:52.363762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:52.363602 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 08:32:53.367903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:53.367868 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:32:53.368418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:53.368398 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:32:53.369063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:53.369041 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:32:53.899095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:53.899057 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:54.273991 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:54.273952 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:54.602590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:54.602493 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:54.993516 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:54.993483 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:55.825458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:55.825414 6160 generic.go:356] "Generic (PLEG): container finished" podID="2a0c8576-8ea7-408d-9a55-aacd9de4a646" containerID="d32bf91552ac51ade316641af73a8b2ec2a3febe8f107e192a901a0030d565c5" exitCode=0 Nov 02 08:32:55.825861 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:55.825492 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" event={"ID":"2a0c8576-8ea7-408d-9a55-aacd9de4a646","Type":"ContainerDied","Data":"d32bf91552ac51ade316641af73a8b2ec2a3febe8f107e192a901a0030d565c5"} Nov 02 08:32:55.900196 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:55.900148 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:32:56.005658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.005623 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0c8576-8ea7-408d-9a55-aacd9de4a646-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0c8576-8ea7-408d-9a55-aacd9de4a646-plugins-config-volume\") pod \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\" (UID: \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\") " Nov 02 08:32:56.005832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.005685 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0c8576-8ea7-408d-9a55-aacd9de4a646-kube-api-access-l4dzs\" (UniqueName: \"kubernetes.io/projected/2a0c8576-8ea7-408d-9a55-aacd9de4a646-kube-api-access-l4dzs\") pod \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\" (UID: \"2a0c8576-8ea7-408d-9a55-aacd9de4a646\") " Nov 02 08:32:56.006121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.006098 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0c8576-8ea7-408d-9a55-aacd9de4a646-plugins-config-volume" pod "2a0c8576-8ea7-408d-9a55-aacd9de4a646" (UID: "2a0c8576-8ea7-408d-9a55-aacd9de4a646"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:32:56.008080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.008063 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0c8576-8ea7-408d-9a55-aacd9de4a646-kube-api-access-l4dzs" pod "2a0c8576-8ea7-408d-9a55-aacd9de4a646" (UID: "2a0c8576-8ea7-408d-9a55-aacd9de4a646"). InnerVolumeSpecName "kube-api-access-l4dzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:32:56.051054 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.051009 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:56.107406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.107374 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0c8576-8ea7-408d-9a55-aacd9de4a646-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:32:56.107406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.107403 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4dzs\" (UniqueName: \"kubernetes.io/projected/2a0c8576-8ea7-408d-9a55-aacd9de4a646-kube-api-access-l4dzs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:32:56.357033 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.356995 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:56.835715 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.835681 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" event={"ID":"2a0c8576-8ea7-408d-9a55-aacd9de4a646","Type":"ContainerDied","Data":"15346df376fc8af5957e4a358451f3b86195eea749e3fd154fd702eba355a324"} Nov 02 08:32:56.836202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.835720 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg" Nov 02 08:32:56.836202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.835751 6160 scope.go:122] "RemoveContainer" containerID="d32bf91552ac51ade316641af73a8b2ec2a3febe8f107e192a901a0030d565c5" Nov 02 08:32:56.996307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:56.996273 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg"] Nov 02 08:32:57.042435 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:57.042401 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cba6f9af-767f954cd7-6ltlg"] Nov 02 08:32:57.588288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:57.588255 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0c8576-8ea7-408d-9a55-aacd9de4a646" path="/var/lib/kubelet/pods/2a0c8576-8ea7-408d-9a55-aacd9de4a646/volumes" Nov 02 08:32:59.689499 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:32:59.689464 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:00.053310 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:00.053215 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:00.313478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:00.313371 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:04.986952 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:04.986915 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:05.226749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:05.226718 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:05.426539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:05.426497 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:05.640947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:05.640899 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:10.117275 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:10.117235 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:10.522156 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:33:10.522113 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:33:10.564318 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:10.564283 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:16.175379 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:16.175339 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:16.429435 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:16.429347 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:16.857349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:16.857263 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:20.447474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:20.447439 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:20.724482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:20.724398 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:21.467667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:21.467613 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:21.769348 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:21.769253 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:23.946778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:23.946744 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:24.148722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.148688 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:24.184745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.184714 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t"] Nov 02 08:33:24.185557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.185533 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2a0c8576-8ea7-408d-9a55-aacd9de4a646" containerName="payload-processing" Nov 02 08:33:24.185557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.185551 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0c8576-8ea7-408d-9a55-aacd9de4a646" containerName="payload-processing" Nov 02 08:33:24.185688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.185678 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0c8576-8ea7-408d-9a55-aacd9de4a646" containerName="payload-processing" Nov 02 08:33:24.189586 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.189564 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:24.195282 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.195256 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpwp\" (UniqueName: \"kubernetes.io/projected/31a63621-ee13-463b-a2c0-50862c92ca21-kube-api-access-qbpwp\") pod \"payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t\" (UID: \"31a63621-ee13-463b-a2c0-50862c92ca21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:24.195425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.195319 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a63621-ee13-463b-a2c0-50862c92ca21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t\" (UID: \"31a63621-ee13-463b-a2c0-50862c92ca21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:24.244190 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.244143 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t"] Nov 02 08:33:24.284544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.284510 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c17125\"" Nov 02 08:33:24.284544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.284510 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c17125-dockercfg-nqm4g\"" Nov 02 08:33:24.296539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.296512 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpwp\" (UniqueName: \"kubernetes.io/projected/31a63621-ee13-463b-a2c0-50862c92ca21-kube-api-access-qbpwp\") pod \"payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t\" (UID: \"31a63621-ee13-463b-a2c0-50862c92ca21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:24.296722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.296611 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a63621-ee13-463b-a2c0-50862c92ca21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t\" (UID: \"31a63621-ee13-463b-a2c0-50862c92ca21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:24.297595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.297562 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a63621-ee13-463b-a2c0-50862c92ca21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t\" (UID: \"31a63621-ee13-463b-a2c0-50862c92ca21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:24.383974 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.383912 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx"] Nov 02 08:33:24.390450 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.390421 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:24.397883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.397850 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpwp\" (UniqueName: \"kubernetes.io/projected/31a63621-ee13-463b-a2c0-50862c92ca21-kube-api-access-qbpwp\") pod \"payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t\" (UID: \"31a63621-ee13-463b-a2c0-50862c92ca21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:24.399007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.398977 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91956a2d-4b70-4d16-85ac-757af9bdf895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx\" (UID: \"91956a2d-4b70-4d16-85ac-757af9bdf895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:24.399282 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.399262 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsg7\" (UniqueName: \"kubernetes.io/projected/91956a2d-4b70-4d16-85ac-757af9bdf895-kube-api-access-mhsg7\") pod \"payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx\" (UID: \"91956a2d-4b70-4d16-85ac-757af9bdf895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:24.443152 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.443119 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx"] Nov 02 08:33:24.501238 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.501096 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91956a2d-4b70-4d16-85ac-757af9bdf895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx\" (UID: \"91956a2d-4b70-4d16-85ac-757af9bdf895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:24.501238 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.501165 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsg7\" (UniqueName: \"kubernetes.io/projected/91956a2d-4b70-4d16-85ac-757af9bdf895-kube-api-access-mhsg7\") pod \"payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx\" (UID: \"91956a2d-4b70-4d16-85ac-757af9bdf895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:24.501885 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.501863 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91956a2d-4b70-4d16-85ac-757af9bdf895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx\" (UID: \"91956a2d-4b70-4d16-85ac-757af9bdf895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:24.503952 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.503934 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:24.558048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.557975 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsg7\" (UniqueName: \"kubernetes.io/projected/91956a2d-4b70-4d16-85ac-757af9bdf895-kube-api-access-mhsg7\") pod \"payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx\" (UID: \"91956a2d-4b70-4d16-85ac-757af9bdf895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:24.745397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.745360 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:24.888237 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:33:24.888188 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a63621_ee13_463b_a2c0_50862c92ca21.slice/crio-6393eebaa673545009e7d1e79c9171ab95adc7c4edf9ead17768dab5244d9bb8 WatchSource:0}: Error finding container 6393eebaa673545009e7d1e79c9171ab95adc7c4edf9ead17768dab5244d9bb8: Status 404 returned error can't find the container with id 6393eebaa673545009e7d1e79c9171ab95adc7c4edf9ead17768dab5244d9bb8 Nov 02 08:33:24.888828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.888755 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t"] Nov 02 08:33:24.917627 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:24.917600 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th"] Nov 02 08:33:25.029235 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:25.029149 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn"] Nov 02 08:33:25.116222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:25.115717 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" event={"ID":"31a63621-ee13-463b-a2c0-50862c92ca21","Type":"ContainerStarted","Data":"86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7"} Nov 02 08:33:25.116222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:25.115786 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" event={"ID":"31a63621-ee13-463b-a2c0-50862c92ca21","Type":"ContainerStarted","Data":"6393eebaa673545009e7d1e79c9171ab95adc7c4edf9ead17768dab5244d9bb8"} Nov 02 08:33:25.116222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:25.116093 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:25.133626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:25.133600 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx"] Nov 02 08:33:25.137511 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:33:25.137470 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91956a2d_4b70_4d16_85ac_757af9bdf895.slice/crio-098036b38f9d7d7092341ee92a2086e954069282582893bbdf6c2354e2d65bba WatchSource:0}: Error finding container 098036b38f9d7d7092341ee92a2086e954069282582893bbdf6c2354e2d65bba: Status 404 returned error can't find the container with id 098036b38f9d7d7092341ee92a2086e954069282582893bbdf6c2354e2d65bba Nov 02 08:33:25.265800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:25.265747 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" podStartSLOduration=1.265731763 podStartE2EDuration="1.265731763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:25.265019667 +0000 UTC m=+1603.818523875" watchObservedRunningTime="2026-11-02 08:33:25.265731763 +0000 UTC m=+1603.819235960" Nov 02 08:33:25.330332 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:25.330236 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:25.545707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:25.545664 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:26.124995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:26.124959 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" event={"ID":"91956a2d-4b70-4d16-85ac-757af9bdf895","Type":"ContainerStarted","Data":"b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39"} Nov 02 08:33:26.125537 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:26.125043 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" event={"ID":"91956a2d-4b70-4d16-85ac-757af9bdf895","Type":"ContainerStarted","Data":"098036b38f9d7d7092341ee92a2086e954069282582893bbdf6c2354e2d65bba"} Nov 02 08:33:26.125537 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:26.125202 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:26.237993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:26.237920 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" podStartSLOduration=2.237907601 podStartE2EDuration="2.237907601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:26.234645786 +0000 UTC m=+1604.788149985" watchObservedRunningTime="2026-11-02 08:33:26.237907601 +0000 UTC m=+1604.791411789" Nov 02 08:33:27.090723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:27.090687 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:27.456424 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:27.456372 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:27.530495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:27.530465 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn"] Nov 02 08:33:28.032746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:28.032674 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:28.211748 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:28.211719 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:29.119116 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:29.119073 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:29.418831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:29.418794 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:29.918198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:29.918126 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" podUID="0fde55a9-ba16-4854-bef3-bd7a05375218" containerName="payload-processing" containerID="cri-o://6e56bc8178bc0614585bb44dbf87184bdff6b2c7acb2b1a63e867ea017b025a0" gracePeriod=25 Nov 02 08:33:30.029611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.029568 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" podUID="ec334a13-fe07-4868-b0b8-e70625372832" containerName="payload-pre-processing" containerID="cri-o://1bfa96e25e52d30967f0493d98c87b8daa9908459fe0440df8487c036e82d08c" gracePeriod=25 Nov 02 08:33:30.170018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.169949 6160 generic.go:356] "Generic (PLEG): container finished" podID="ec334a13-fe07-4868-b0b8-e70625372832" containerID="1bfa96e25e52d30967f0493d98c87b8daa9908459fe0440df8487c036e82d08c" exitCode=0 Nov 02 08:33:30.170429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.170030 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" event={"ID":"ec334a13-fe07-4868-b0b8-e70625372832","Type":"ContainerDied","Data":"1bfa96e25e52d30967f0493d98c87b8daa9908459fe0440df8487c036e82d08c"} Nov 02 08:33:30.172056 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.171939 6160 generic.go:356] "Generic (PLEG): container finished" podID="0fde55a9-ba16-4854-bef3-bd7a05375218" containerID="6e56bc8178bc0614585bb44dbf87184bdff6b2c7acb2b1a63e867ea017b025a0" exitCode=0 Nov 02 08:33:30.172159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.172127 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" event={"ID":"0fde55a9-ba16-4854-bef3-bd7a05375218","Type":"ContainerDied","Data":"6e56bc8178bc0614585bb44dbf87184bdff6b2c7acb2b1a63e867ea017b025a0"} Nov 02 08:33:30.210194 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.209799 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:33:30.291885 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.291826 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fde55a9-ba16-4854-bef3-bd7a05375218-kube-api-access-td6f2\" (UniqueName: \"kubernetes.io/projected/0fde55a9-ba16-4854-bef3-bd7a05375218-kube-api-access-td6f2\") pod \"0fde55a9-ba16-4854-bef3-bd7a05375218\" (UID: \"0fde55a9-ba16-4854-bef3-bd7a05375218\") " Nov 02 08:33:30.292119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.292096 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fde55a9-ba16-4854-bef3-bd7a05375218-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fde55a9-ba16-4854-bef3-bd7a05375218-plugins-config-volume\") pod \"0fde55a9-ba16-4854-bef3-bd7a05375218\" (UID: \"0fde55a9-ba16-4854-bef3-bd7a05375218\") " Nov 02 08:33:30.292950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.292900 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fde55a9-ba16-4854-bef3-bd7a05375218-plugins-config-volume" pod "0fde55a9-ba16-4854-bef3-bd7a05375218" (UID: "0fde55a9-ba16-4854-bef3-bd7a05375218"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:33:30.296874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.296849 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fde55a9-ba16-4854-bef3-bd7a05375218-kube-api-access-td6f2" pod "0fde55a9-ba16-4854-bef3-bd7a05375218" (UID: "0fde55a9-ba16-4854-bef3-bd7a05375218"). InnerVolumeSpecName "kube-api-access-td6f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:33:30.328776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.328750 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:33:30.393506 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.393470 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec334a13-fe07-4868-b0b8-e70625372832-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec334a13-fe07-4868-b0b8-e70625372832-plugins-config-volume\") pod \"ec334a13-fe07-4868-b0b8-e70625372832\" (UID: \"ec334a13-fe07-4868-b0b8-e70625372832\") " Nov 02 08:33:30.393682 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.393661 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec334a13-fe07-4868-b0b8-e70625372832-kube-api-access-xtt2p\" (UniqueName: \"kubernetes.io/projected/ec334a13-fe07-4868-b0b8-e70625372832-kube-api-access-xtt2p\") pod \"ec334a13-fe07-4868-b0b8-e70625372832\" (UID: \"ec334a13-fe07-4868-b0b8-e70625372832\") " Nov 02 08:33:30.393928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.393906 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec334a13-fe07-4868-b0b8-e70625372832-plugins-config-volume" pod "ec334a13-fe07-4868-b0b8-e70625372832" (UID: "ec334a13-fe07-4868-b0b8-e70625372832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:33:30.394244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.394227 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td6f2\" (UniqueName: \"kubernetes.io/projected/0fde55a9-ba16-4854-bef3-bd7a05375218-kube-api-access-td6f2\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:30.394287 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.394246 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fde55a9-ba16-4854-bef3-bd7a05375218-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:30.394287 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.394255 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec334a13-fe07-4868-b0b8-e70625372832-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:30.395895 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.395872 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec334a13-fe07-4868-b0b8-e70625372832-kube-api-access-xtt2p" pod "ec334a13-fe07-4868-b0b8-e70625372832" (UID: "ec334a13-fe07-4868-b0b8-e70625372832"). InnerVolumeSpecName "kube-api-access-xtt2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:33:30.495581 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:30.495493 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtt2p\" (UniqueName: \"kubernetes.io/projected/ec334a13-fe07-4868-b0b8-e70625372832-kube-api-access-xtt2p\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:31.186340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.186228 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" event={"ID":"ec334a13-fe07-4868-b0b8-e70625372832","Type":"ContainerDied","Data":"aa74cb8f7f74f535c6bc948f3e77b73e3f723ed0e10e32fbb40c0ece184de199"} Nov 02 08:33:31.186340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.186307 6160 scope.go:122] "RemoveContainer" containerID="1bfa96e25e52d30967f0493d98c87b8daa9908459fe0440df8487c036e82d08c" Nov 02 08:33:31.186340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.186242 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn" Nov 02 08:33:31.191921 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.191893 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" Nov 02 08:33:31.192211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.192144 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th" event={"ID":"0fde55a9-ba16-4854-bef3-bd7a05375218","Type":"ContainerDied","Data":"5cbfd354354cd97db2076bd0b978de8c4d98cc855c5b025b894281bab416f245"} Nov 02 08:33:31.203557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.203531 6160 scope.go:122] "RemoveContainer" containerID="6e56bc8178bc0614585bb44dbf87184bdff6b2c7acb2b1a63e867ea017b025a0" Nov 02 08:33:31.345438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.345407 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th"] Nov 02 08:33:31.351677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.351652 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5bb4f-697d79c597-jx6th"] Nov 02 08:33:31.457346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.457263 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn"] Nov 02 08:33:31.463635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.463602 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5bb4f-76dd5cc59f-gzrmn"] Nov 02 08:33:31.589009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.588978 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fde55a9-ba16-4854-bef3-bd7a05375218" path="/var/lib/kubelet/pods/0fde55a9-ba16-4854-bef3-bd7a05375218/volumes" Nov 02 08:33:31.589484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.589467 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec334a13-fe07-4868-b0b8-e70625372832" path="/var/lib/kubelet/pods/ec334a13-fe07-4868-b0b8-e70625372832/volumes" Nov 02 08:33:31.721550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:31.721442 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:32.531381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:32.531340 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" podUID="dc93aac4-9fbd-4739-a09c-be525981d22b" containerName="payload-processing" containerID="cri-o://219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71" gracePeriod=25 Nov 02 08:33:32.799519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:32.799452 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:33:32.927648 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:32.927606 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc93aac4-9fbd-4739-a09c-be525981d22b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc93aac4-9fbd-4739-a09c-be525981d22b-plugins-config-volume\") pod \"dc93aac4-9fbd-4739-a09c-be525981d22b\" (UID: \"dc93aac4-9fbd-4739-a09c-be525981d22b\") " Nov 02 08:33:32.927818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:32.927663 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc93aac4-9fbd-4739-a09c-be525981d22b-kube-api-access-6jzwr\" (UniqueName: \"kubernetes.io/projected/dc93aac4-9fbd-4739-a09c-be525981d22b-kube-api-access-6jzwr\") pod \"dc93aac4-9fbd-4739-a09c-be525981d22b\" (UID: \"dc93aac4-9fbd-4739-a09c-be525981d22b\") " Nov 02 08:33:32.927984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:32.927958 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc93aac4-9fbd-4739-a09c-be525981d22b-plugins-config-volume" pod "dc93aac4-9fbd-4739-a09c-be525981d22b" (UID: "dc93aac4-9fbd-4739-a09c-be525981d22b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:33:32.928497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:32.928393 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc93aac4-9fbd-4739-a09c-be525981d22b-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:32.930193 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:32.930147 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc93aac4-9fbd-4739-a09c-be525981d22b-kube-api-access-6jzwr" pod "dc93aac4-9fbd-4739-a09c-be525981d22b" (UID: "dc93aac4-9fbd-4739-a09c-be525981d22b"). InnerVolumeSpecName "kube-api-access-6jzwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:33:33.030150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.030113 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzwr\" (UniqueName: \"kubernetes.io/projected/dc93aac4-9fbd-4739-a09c-be525981d22b-kube-api-access-6jzwr\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:33:33.061067 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.060986 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:33.214581 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.214548 6160 generic.go:356] "Generic (PLEG): container finished" podID="dc93aac4-9fbd-4739-a09c-be525981d22b" containerID="219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71" exitCode=0 Nov 02 08:33:33.214754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.214594 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" Nov 02 08:33:33.214754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.214632 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" event={"ID":"dc93aac4-9fbd-4739-a09c-be525981d22b","Type":"ContainerDied","Data":"219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71"} Nov 02 08:33:33.214754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.214684 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn" event={"ID":"dc93aac4-9fbd-4739-a09c-be525981d22b","Type":"ContainerDied","Data":"e6c3c1fa1beb84e5fd85b8b927c447202d50a9bcc8d16f4379786d7653dafc0f"} Nov 02 08:33:33.214754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.214707 6160 scope.go:122] "RemoveContainer" containerID="219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71" Nov 02 08:33:33.225997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.225981 6160 scope.go:122] "RemoveContainer" containerID="219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71" Nov 02 08:33:33.226269 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:33:33.226249 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71\": container with ID starting with 219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71 not found: ID does not exist" containerID="219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71" Nov 02 08:33:33.226339 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.226282 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71"} err="failed to get container status \"219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71\": rpc error: code = NotFound desc = could not find container \"219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71\": container with ID starting with 219f75813bbd2b1be7498ee398361fadb22cf5555587033b5168fb1557317c71 not found: ID does not exist" Nov 02 08:33:33.379455 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.379362 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn"] Nov 02 08:33:33.383466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.383440 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19ee94-5b695cbd99-jmxjn"] Nov 02 08:33:33.587528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.587495 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc93aac4-9fbd-4739-a09c-be525981d22b" path="/var/lib/kubelet/pods/dc93aac4-9fbd-4739-a09c-be525981d22b/volumes" Nov 02 08:33:33.832144 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:33.832109 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:34.200318 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:34.200283 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:34.445807 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:34.445778 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:36.128881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.128853 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:33:36.875076 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875040 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t"] Nov 02 08:33:36.875735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875721 6160 cpu_manager.go:414] "containerMap: removing container" podUID="dc93aac4-9fbd-4739-a09c-be525981d22b" containerName="payload-processing" Nov 02 08:33:36.875784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875736 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc93aac4-9fbd-4739-a09c-be525981d22b" containerName="payload-processing" Nov 02 08:33:36.875784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875761 6160 cpu_manager.go:414] "containerMap: removing container" podUID="0fde55a9-ba16-4854-bef3-bd7a05375218" containerName="payload-processing" Nov 02 08:33:36.875784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875767 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fde55a9-ba16-4854-bef3-bd7a05375218" containerName="payload-processing" Nov 02 08:33:36.875883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875800 6160 cpu_manager.go:414] "containerMap: removing container" podUID="ec334a13-fe07-4868-b0b8-e70625372832" containerName="payload-pre-processing" Nov 02 08:33:36.875883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875805 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec334a13-fe07-4868-b0b8-e70625372832" containerName="payload-pre-processing" Nov 02 08:33:36.875940 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875894 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec334a13-fe07-4868-b0b8-e70625372832" containerName="payload-pre-processing" Nov 02 08:33:36.875940 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875901 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc93aac4-9fbd-4739-a09c-be525981d22b" containerName="payload-processing" Nov 02 08:33:36.875940 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.875911 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fde55a9-ba16-4854-bef3-bd7a05375218" containerName="payload-processing" Nov 02 08:33:36.880945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.880927 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.930007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.929978 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-69a9bb-openshift-default-dockercfg-6vf2g\"" Nov 02 08:33:36.980987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.980948 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.981217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.981037 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.981217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.981074 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.981217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.981101 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.981418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.981233 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.981418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.981382 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a90a9e-68c6-44e6-b25f-4697524795f6-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.981538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.981439 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.981538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.981473 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.981538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.981509 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htn6k\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-kube-api-access-htn6k\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:36.982247 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:36.982222 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t"] Nov 02 08:33:37.083042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083010 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a90a9e-68c6-44e6-b25f-4697524795f6-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083061 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083080 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083206 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htn6k\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-kube-api-access-htn6k\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083342 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083275 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083342 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083330 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083363 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083389 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083496 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083512 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083732 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083840 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083792 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.083880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.083861 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.084072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.084051 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a90a9e-68c6-44e6-b25f-4697524795f6-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.085620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.085604 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.085847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.085829 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.138475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.138382 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:33:37.139053 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.138746 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.143892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.143861 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htn6k\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-kube-api-access-htn6k\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.195435 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.195408 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:37.640500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.640473 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t"] Nov 02 08:33:37.641155 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:33:37.641117 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a90a9e_68c6_44e6_b25f_4697524795f6.slice/crio-999c336a7acb4b5f3c7870ce57fa748edf700eb8f1783b09fbe84830e94a052e WatchSource:0}: Error finding container 999c336a7acb4b5f3c7870ce57fa748edf700eb8f1783b09fbe84830e94a052e: Status 404 returned error can't find the container with id 999c336a7acb4b5f3c7870ce57fa748edf700eb8f1783b09fbe84830e94a052e Nov 02 08:33:37.644250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.644209 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:33:37.644342 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.644299 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:33:37.644381 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:37.644343 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:33:38.260120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:38.259503 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" event={"ID":"95a90a9e-68c6-44e6-b25f-4697524795f6","Type":"ContainerStarted","Data":"7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f"} Nov 02 08:33:38.260120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:38.259565 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" event={"ID":"95a90a9e-68c6-44e6-b25f-4697524795f6","Type":"ContainerStarted","Data":"999c336a7acb4b5f3c7870ce57fa748edf700eb8f1783b09fbe84830e94a052e"} Nov 02 08:33:38.376075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:38.375998 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" podStartSLOduration=2.375974824 podStartE2EDuration="2.375974824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:38.373714384 +0000 UTC m=+1616.927218584" watchObservedRunningTime="2026-11-02 08:33:38.375974824 +0000 UTC m=+1616.929479021" Nov 02 08:33:38.553615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:38.553525 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:39.159575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.159540 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:39.196035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.196009 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:39.197458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.197421 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 08:33:39.197577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.197473 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 08:33:39.361216 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.350190 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:39.557106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.557053 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk"] Nov 02 08:33:39.562295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.562268 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:39.645829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.645792 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk"] Nov 02 08:33:39.659674 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.659573 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f3da8a-b765-44b0-b28d-c20f48cbf732-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk\" (UID: \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:39.659835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.659814 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txjs\" (UniqueName: \"kubernetes.io/projected/c1f3da8a-b765-44b0-b28d-c20f48cbf732-kube-api-access-2txjs\") pod \"payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk\" (UID: \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:39.665352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.665322 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b6a289-dockercfg-sqqqn\"" Nov 02 08:33:39.665498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.665404 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b6a289\"" Nov 02 08:33:39.751322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.749954 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769"] Nov 02 08:33:39.777957 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.761280 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txjs\" (UniqueName: \"kubernetes.io/projected/c1f3da8a-b765-44b0-b28d-c20f48cbf732-kube-api-access-2txjs\") pod \"payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk\" (UID: \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:39.778473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.778438 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f3da8a-b765-44b0-b28d-c20f48cbf732-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk\" (UID: \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:39.781096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.781069 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f3da8a-b765-44b0-b28d-c20f48cbf732-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk\" (UID: \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:39.797493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.797458 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769"] Nov 02 08:33:39.797766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.797747 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:39.863210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.862341 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txjs\" (UniqueName: \"kubernetes.io/projected/c1f3da8a-b765-44b0-b28d-c20f48cbf732-kube-api-access-2txjs\") pod \"payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk\" (UID: \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:39.880653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.880610 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5nj5\" (UniqueName: \"kubernetes.io/projected/570f3f22-3c1d-4206-b0dd-474023f4e48a-kube-api-access-t5nj5\") pod \"payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769\" (UID: \"570f3f22-3c1d-4206-b0dd-474023f4e48a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:39.881140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.881118 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f3f22-3c1d-4206-b0dd-474023f4e48a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769\" (UID: \"570f3f22-3c1d-4206-b0dd-474023f4e48a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:39.885875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.885670 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:39.983799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.983775 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5nj5\" (UniqueName: \"kubernetes.io/projected/570f3f22-3c1d-4206-b0dd-474023f4e48a-kube-api-access-t5nj5\") pod \"payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769\" (UID: \"570f3f22-3c1d-4206-b0dd-474023f4e48a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:39.984458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.983982 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f3f22-3c1d-4206-b0dd-474023f4e48a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769\" (UID: \"570f3f22-3c1d-4206-b0dd-474023f4e48a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:39.985024 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:39.984951 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f3f22-3c1d-4206-b0dd-474023f4e48a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769\" (UID: \"570f3f22-3c1d-4206-b0dd-474023f4e48a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:40.041222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:40.041092 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5nj5\" (UniqueName: \"kubernetes.io/projected/570f3f22-3c1d-4206-b0dd-474023f4e48a-kube-api-access-t5nj5\") pod \"payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769\" (UID: \"570f3f22-3c1d-4206-b0dd-474023f4e48a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:40.160945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:40.160843 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:40.196717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:40.196689 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 08:33:40.196861 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:40.196740 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 08:33:40.297485 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:40.297453 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk"] Nov 02 08:33:40.544101 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:33:40.544066 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:33:40.566699 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:40.566669 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769"] Nov 02 08:33:40.567602 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:33:40.567574 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570f3f22_3c1d_4206_b0dd_474023f4e48a.slice/crio-21b0a68138933576d64899fb06d733caff1e322d6d15b852eb52e77e59f00343 WatchSource:0}: Error finding container 21b0a68138933576d64899fb06d733caff1e322d6d15b852eb52e77e59f00343: Status 404 returned error can't find the container with id 21b0a68138933576d64899fb06d733caff1e322d6d15b852eb52e77e59f00343 Nov 02 08:33:41.195903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.195867 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 08:33:41.196085 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.195929 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 08:33:41.291548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.291508 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" event={"ID":"570f3f22-3c1d-4206-b0dd-474023f4e48a","Type":"ContainerStarted","Data":"7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88"} Nov 02 08:33:41.291695 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.291591 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" event={"ID":"570f3f22-3c1d-4206-b0dd-474023f4e48a","Type":"ContainerStarted","Data":"21b0a68138933576d64899fb06d733caff1e322d6d15b852eb52e77e59f00343"} Nov 02 08:33:41.291848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.291832 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:41.293359 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.293337 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" event={"ID":"c1f3da8a-b765-44b0-b28d-c20f48cbf732","Type":"ContainerStarted","Data":"70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb"} Nov 02 08:33:41.293515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.293503 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" event={"ID":"c1f3da8a-b765-44b0-b28d-c20f48cbf732","Type":"ContainerStarted","Data":"261ff3e54d140a13d139b5e0fc5d58dab786177376a6857cbcf4629598545e8f"} Nov 02 08:33:41.293607 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.293596 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:41.430201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.426972 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" podStartSLOduration=2.42695029 podStartE2EDuration="2.42695029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:41.420702531 +0000 UTC m=+1619.974206731" watchObservedRunningTime="2026-11-02 08:33:41.42695029 +0000 UTC m=+1619.980454492" Nov 02 08:33:41.538345 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:41.534763 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" podStartSLOduration=2.5347444709999998 podStartE2EDuration="2.534744471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:41.533670628 +0000 UTC m=+1620.087174822" watchObservedRunningTime="2026-11-02 08:33:41.534744471 +0000 UTC m=+1620.088248669" Nov 02 08:33:42.195949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:42.195775 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 08:33:42.195949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:42.195828 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 08:33:43.196022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:43.195983 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 08:33:43.196523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:43.196045 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 08:33:44.200255 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:44.200226 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:44.200753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:44.200734 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:44.201557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:44.201535 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:33:44.621760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:44.621670 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:46.180057 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:46.180022 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:49.922576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:49.922537 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:52.304926 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:52.304890 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:33:52.305563 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:52.305509 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:33:54.820501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:54.820460 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:59.970250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:33:59.970212 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:05.197764 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:05.197730 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:09.927402 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:09.927368 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4"] Nov 02 08:34:10.045674 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:10.045633 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9"] Nov 02 08:34:10.250219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:10.250107 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:10.487918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:10.487881 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:10.555648 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:34:10.555549 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:34:10.650478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:10.650411 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:12.029422 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:12.029391 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:12.781703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:12.781664 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:13.874822 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:13.874786 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:14.371954 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:14.371922 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:14.928655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:14.928614 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" podUID="037a02af-4aa5-4abe-9d4b-3659f87869b0" containerName="payload-processing" containerID="cri-o://c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10" gracePeriod=25 Nov 02 08:34:15.046413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.046372 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" podUID="004c0899-355d-45d5-8cff-743aba69ff08" containerName="payload-pre-processing" containerID="cri-o://3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32" gracePeriod=25 Nov 02 08:34:15.219079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.218802 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:34:15.320923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.320891 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037a02af-4aa5-4abe-9d4b-3659f87869b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037a02af-4aa5-4abe-9d4b-3659f87869b0-plugins-config-volume\") pod \"037a02af-4aa5-4abe-9d4b-3659f87869b0\" (UID: \"037a02af-4aa5-4abe-9d4b-3659f87869b0\") " Nov 02 08:34:15.321121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.321092 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037a02af-4aa5-4abe-9d4b-3659f87869b0-kube-api-access-wg6rk\" (UniqueName: \"kubernetes.io/projected/037a02af-4aa5-4abe-9d4b-3659f87869b0-kube-api-access-wg6rk\") pod \"037a02af-4aa5-4abe-9d4b-3659f87869b0\" (UID: \"037a02af-4aa5-4abe-9d4b-3659f87869b0\") " Nov 02 08:34:15.321298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.321265 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037a02af-4aa5-4abe-9d4b-3659f87869b0-plugins-config-volume" pod "037a02af-4aa5-4abe-9d4b-3659f87869b0" (UID: "037a02af-4aa5-4abe-9d4b-3659f87869b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:34:15.321640 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.321618 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037a02af-4aa5-4abe-9d4b-3659f87869b0-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:15.323592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.323564 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037a02af-4aa5-4abe-9d4b-3659f87869b0-kube-api-access-wg6rk" pod "037a02af-4aa5-4abe-9d4b-3659f87869b0" (UID: "037a02af-4aa5-4abe-9d4b-3659f87869b0"). InnerVolumeSpecName "kube-api-access-wg6rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:15.326304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.326285 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:34:15.422975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.422947 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004c0899-355d-45d5-8cff-743aba69ff08-kube-api-access-f9ht5\" (UniqueName: \"kubernetes.io/projected/004c0899-355d-45d5-8cff-743aba69ff08-kube-api-access-f9ht5\") pod \"004c0899-355d-45d5-8cff-743aba69ff08\" (UID: \"004c0899-355d-45d5-8cff-743aba69ff08\") " Nov 02 08:34:15.423139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.423084 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004c0899-355d-45d5-8cff-743aba69ff08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004c0899-355d-45d5-8cff-743aba69ff08-plugins-config-volume\") pod \"004c0899-355d-45d5-8cff-743aba69ff08\" (UID: \"004c0899-355d-45d5-8cff-743aba69ff08\") " Nov 02 08:34:15.423484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.423452 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004c0899-355d-45d5-8cff-743aba69ff08-plugins-config-volume" pod "004c0899-355d-45d5-8cff-743aba69ff08" (UID: "004c0899-355d-45d5-8cff-743aba69ff08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:34:15.423484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.423463 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg6rk\" (UniqueName: \"kubernetes.io/projected/037a02af-4aa5-4abe-9d4b-3659f87869b0-kube-api-access-wg6rk\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:15.425379 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.425355 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004c0899-355d-45d5-8cff-743aba69ff08-kube-api-access-f9ht5" pod "004c0899-355d-45d5-8cff-743aba69ff08" (UID: "004c0899-355d-45d5-8cff-743aba69ff08"). InnerVolumeSpecName "kube-api-access-f9ht5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:15.453524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.453433 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698"] Nov 02 08:34:15.455398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.455369 6160 cpu_manager.go:414] "containerMap: removing container" podUID="004c0899-355d-45d5-8cff-743aba69ff08" containerName="payload-pre-processing" Nov 02 08:34:15.455556 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.455541 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004c0899-355d-45d5-8cff-743aba69ff08" containerName="payload-pre-processing" Nov 02 08:34:15.455650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.455640 6160 cpu_manager.go:414] "containerMap: removing container" podUID="037a02af-4aa5-4abe-9d4b-3659f87869b0" containerName="payload-processing" Nov 02 08:34:15.455732 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.455721 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037a02af-4aa5-4abe-9d4b-3659f87869b0" containerName="payload-processing" Nov 02 08:34:15.456130 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.456112 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="004c0899-355d-45d5-8cff-743aba69ff08" containerName="payload-pre-processing" Nov 02 08:34:15.456272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.456260 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="037a02af-4aa5-4abe-9d4b-3659f87869b0" containerName="payload-processing" Nov 02 08:34:15.461729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.461707 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:15.511469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.511435 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bca68b\"" Nov 02 08:34:15.511614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.511573 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bca68b-dockercfg-b9k5k\"" Nov 02 08:34:15.524600 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.524570 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ht5\" (UniqueName: \"kubernetes.io/projected/004c0899-355d-45d5-8cff-743aba69ff08-kube-api-access-f9ht5\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:15.524731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.524604 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004c0899-355d-45d5-8cff-743aba69ff08-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:15.545215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.545187 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698"] Nov 02 08:34:15.603654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.603628 6160 generic.go:356] "Generic (PLEG): container finished" podID="037a02af-4aa5-4abe-9d4b-3659f87869b0" containerID="c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10" exitCode=0 Nov 02 08:34:15.603788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.603754 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" Nov 02 08:34:15.603788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.603763 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" event={"ID":"037a02af-4aa5-4abe-9d4b-3659f87869b0","Type":"ContainerDied","Data":"c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10"} Nov 02 08:34:15.603901 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.603822 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4" event={"ID":"037a02af-4aa5-4abe-9d4b-3659f87869b0","Type":"ContainerDied","Data":"d6c7272ab57b0ffaf5c2ecb4aca8d7541614c33d85fce76a9e6f6fe827ba3626"} Nov 02 08:34:15.603901 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.603854 6160 scope.go:122] "RemoveContainer" containerID="c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10" Nov 02 08:34:15.605315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.605290 6160 generic.go:356] "Generic (PLEG): container finished" podID="004c0899-355d-45d5-8cff-743aba69ff08" containerID="3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32" exitCode=0 Nov 02 08:34:15.605436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.605341 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" Nov 02 08:34:15.605436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.605375 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" event={"ID":"004c0899-355d-45d5-8cff-743aba69ff08","Type":"ContainerDied","Data":"3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32"} Nov 02 08:34:15.605436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.605414 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9" event={"ID":"004c0899-355d-45d5-8cff-743aba69ff08","Type":"ContainerDied","Data":"96af192a544d4ddc6c3683bbcaa483bd39ae1a2c1e0d3ffa8480f13c8e95be93"} Nov 02 08:34:15.615948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.615931 6160 scope.go:122] "RemoveContainer" containerID="c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10" Nov 02 08:34:15.616237 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:34:15.616215 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10\": container with ID starting with c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10 not found: ID does not exist" containerID="c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10" Nov 02 08:34:15.616304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.616245 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10"} err="failed to get container status \"c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10\": rpc error: code = NotFound desc = could not find container \"c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10\": container with ID starting with c0a609b47680c85c528cae9110484e697b1704c8cee24cc6ce71bf118627da10 not found: ID does not exist" Nov 02 08:34:15.616304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.616263 6160 scope.go:122] "RemoveContainer" containerID="3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32" Nov 02 08:34:15.626106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.626088 6160 scope.go:122] "RemoveContainer" containerID="3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32" Nov 02 08:34:15.626162 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.626109 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4add2425-f901-4880-bb0e-a74a7d44d33e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698\" (UID: \"4add2425-f901-4880-bb0e-a74a7d44d33e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:15.626162 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.626133 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nvv\" (UniqueName: \"kubernetes.io/projected/4add2425-f901-4880-bb0e-a74a7d44d33e-kube-api-access-z4nvv\") pod \"payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698\" (UID: \"4add2425-f901-4880-bb0e-a74a7d44d33e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:15.626413 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:34:15.626394 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32\": container with ID starting with 3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32 not found: ID does not exist" containerID="3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32" Nov 02 08:34:15.626460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.626420 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32"} err="failed to get container status \"3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32\": rpc error: code = NotFound desc = could not find container \"3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32\": container with ID starting with 3886b4063ca5baa3e3a3c183c5ce73505a521601d78f31936aa82fbca1f1ea32 not found: ID does not exist" Nov 02 08:34:15.727473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.727388 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4add2425-f901-4880-bb0e-a74a7d44d33e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698\" (UID: \"4add2425-f901-4880-bb0e-a74a7d44d33e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:15.727473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.727421 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nvv\" (UniqueName: \"kubernetes.io/projected/4add2425-f901-4880-bb0e-a74a7d44d33e-kube-api-access-z4nvv\") pod \"payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698\" (UID: \"4add2425-f901-4880-bb0e-a74a7d44d33e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:15.728150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.728124 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4add2425-f901-4880-bb0e-a74a7d44d33e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698\" (UID: \"4add2425-f901-4880-bb0e-a74a7d44d33e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:15.764302 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.764224 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4"] Nov 02 08:34:15.788203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.785116 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b26f03-5cc97cdb97-dq8b4"] Nov 02 08:34:15.792404 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.792375 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nvv\" (UniqueName: \"kubernetes.io/projected/4add2425-f901-4880-bb0e-a74a7d44d33e-kube-api-access-z4nvv\") pod \"payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698\" (UID: \"4add2425-f901-4880-bb0e-a74a7d44d33e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:15.869513 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.869479 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9"] Nov 02 08:34:15.875035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:15.875005 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b26f03-97b5f4d87-x84v9"] Nov 02 08:34:16.080141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.080059 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:16.458429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.458395 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698"] Nov 02 08:34:16.458841 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:34:16.458808 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4add2425_f901_4880_bb0e_a74a7d44d33e.slice/crio-4eaecf068c45d84db89cce4232fb36b398aabd56d47b0e363a95ddded60156db WatchSource:0}: Error finding container 4eaecf068c45d84db89cce4232fb36b398aabd56d47b0e363a95ddded60156db: Status 404 returned error can't find the container with id 4eaecf068c45d84db89cce4232fb36b398aabd56d47b0e363a95ddded60156db Nov 02 08:34:16.616231 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.616196 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" event={"ID":"4add2425-f901-4880-bb0e-a74a7d44d33e","Type":"ContainerStarted","Data":"4eaecf068c45d84db89cce4232fb36b398aabd56d47b0e363a95ddded60156db"} Nov 02 08:34:16.674624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.674592 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8"] Nov 02 08:34:16.680625 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.680599 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.730234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.730108 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b9eea3-serving-cert\"" Nov 02 08:34:16.734608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.734579 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8"] Nov 02 08:34:16.842705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.842671 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.842892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.842803 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsv89\" (UniqueName: \"kubernetes.io/projected/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-kube-api-access-hsv89\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.842956 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.842936 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-metrics-certs\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.843130 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.843104 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.886046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.886006 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:16.944128 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.944091 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.944352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.944161 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv89\" (UniqueName: \"kubernetes.io/projected/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-kube-api-access-hsv89\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.944352 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:34:16.944256 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-serving-cert: secret "maas-api-e2e-api-b-b9eea3-serving-cert" not found Nov 02 08:34:16.944352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.944272 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-metrics-certs\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.944352 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:34:16.944324 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls podName:c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc nodeName:}" failed. No retries permitted until 2026-11-02 08:34:17.444305106 +0000 UTC m=+1655.997809293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls") pod "maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" (UID: "c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc") : secret "maas-api-e2e-api-b-b9eea3-serving-cert" not found Nov 02 08:34:16.944530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.944423 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.945817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.945784 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.947223 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.947198 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-metrics-certs\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:16.998710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:16.998634 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsv89\" (UniqueName: \"kubernetes.io/projected/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-kube-api-access-hsv89\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:17.055774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:17.055741 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:17.345221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:17.345107 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:17.449907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:17.449869 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:17.452544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:17.452511 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls\") pod \"maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:17.591857 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:17.591826 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004c0899-355d-45d5-8cff-743aba69ff08" path="/var/lib/kubelet/pods/004c0899-355d-45d5-8cff-743aba69ff08/volumes" Nov 02 08:34:17.592295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:17.592280 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037a02af-4aa5-4abe-9d4b-3659f87869b0" path="/var/lib/kubelet/pods/037a02af-4aa5-4abe-9d4b-3659f87869b0/volumes" Nov 02 08:34:17.596970 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:17.596915 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:17.990372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:17.990333 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8"] Nov 02 08:34:18.638070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:18.637980 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" event={"ID":"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc","Type":"ContainerStarted","Data":"c8905d4b95eab104f70900f530f1e07286af34eb32f2983fde523e711ced739b"} Nov 02 08:34:18.639675 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:18.639647 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" event={"ID":"4add2425-f901-4880-bb0e-a74a7d44d33e","Type":"ContainerStarted","Data":"2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb"} Nov 02 08:34:18.639808 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:18.639783 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:18.749330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:18.749278 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" podStartSLOduration=1.826650428 podStartE2EDuration="3.74926224s" totalImagesPullingTime="1.922611812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:18.746301456 +0000 UTC m=+1657.299805654" watchObservedRunningTime="2026-11-02 08:34:18.74926224 +0000 UTC m=+1657.302766437" Nov 02 08:34:19.652778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:19.652739 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" event={"ID":"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc","Type":"ContainerStarted","Data":"c6ef9174351e55a15dc58d7c2fc59ca1be2e5bb82d226c457366dd978995ee2d"} Nov 02 08:34:19.653283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:19.652810 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:19.766961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:19.766901 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" podStartSLOduration=2.732197275 podStartE2EDuration="3.766803471s" totalImagesPullingTime="1.034606196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:19.760538264 +0000 UTC m=+1658.314042461" watchObservedRunningTime="2026-11-02 08:34:19.766803471 +0000 UTC m=+1658.320307684" Nov 02 08:34:21.766079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:21.766044 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:22.028406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.028307 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:22.226490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.226459 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4"] Nov 02 08:34:22.239923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.239895 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.328036 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.327942 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.328383 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.328337 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.328511 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.328426 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.328511 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.328485 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.328631 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.328584 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.328631 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.328608 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxtk\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-kube-api-access-cxxtk\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.328751 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.328654 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.328751 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.328692 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.329228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.328797 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.430484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.430447 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.430708 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.430565 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.430708 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.430678 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.430788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.430770 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.430821 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.430810 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.430868 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.430853 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.430969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.430948 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.431150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.431131 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxtk\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-kube-api-access-cxxtk\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.431326 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.431309 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.431999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.431969 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.432105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.431461 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.432193 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.432116 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.432452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.432405 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.432743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.432722 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.436069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.436029 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.436575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.436550 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.438066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.438039 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4"] Nov 02 08:34:22.484482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.484452 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.484619 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.484531 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxtk\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-kube-api-access-cxxtk\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.556361 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.556329 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:22.988062 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:34:22.988022 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfaa8ae_8d03_4f3e_833a_39b9ba979100.slice/crio-0bfbe862119e4416bad35332157befe1567c365aa7a9300be14b0b9e82646726 WatchSource:0}: Error finding container 0bfbe862119e4416bad35332157befe1567c365aa7a9300be14b0b9e82646726: Status 404 returned error can't find the container with id 0bfbe862119e4416bad35332157befe1567c365aa7a9300be14b0b9e82646726 Nov 02 08:34:22.988564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.988113 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4"] Nov 02 08:34:22.990957 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.990920 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:34:22.991061 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.990992 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:34:22.991061 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:22.991022 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:34:23.686708 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:23.686668 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" event={"ID":"5bfaa8ae-8d03-4f3e-833a-39b9ba979100","Type":"ContainerStarted","Data":"a31639e9114b7ca946d277529f9c7cbcf8f57de9cf71a29d28fa04204df59ee9"} Nov 02 08:34:23.687008 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:23.686732 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" event={"ID":"5bfaa8ae-8d03-4f3e-833a-39b9ba979100","Type":"ContainerStarted","Data":"0bfbe862119e4416bad35332157befe1567c365aa7a9300be14b0b9e82646726"} Nov 02 08:34:23.843091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:23.842963 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" podStartSLOduration=1.842920638 podStartE2EDuration="1.842920638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:23.842590296 +0000 UTC m=+1662.396094488" watchObservedRunningTime="2026-11-02 08:34:23.842920638 +0000 UTC m=+1662.396424837" Nov 02 08:34:24.556690 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:24.556651 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:24.558671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:24.558649 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 08:34:24.558733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:24.558696 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 08:34:25.556797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:25.556750 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 08:34:25.557271 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:25.556815 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 08:34:25.666670 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:25.666641 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:34:26.557774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:26.557732 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 08:34:26.558219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:26.557797 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 08:34:27.071021 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:27.070989 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:27.557704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:27.557665 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 08:34:27.557912 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:27.557747 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 08:34:28.556871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:28.556831 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 08:34:28.557048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:28.556887 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 08:34:29.561680 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:29.561644 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:29.562212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:29.562030 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:29.562629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:29.562607 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:34:29.656124 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:29.656094 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:34:31.943962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:31.943922 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:32.101099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:32.101065 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:36.703772 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:36.703733 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:36.906658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:36.906625 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:37.066403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.066313 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:37.325797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.325710 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc"] Nov 02 08:34:37.330319 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.330289 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.369333 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.369304 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.369530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.369343 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.369530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.369363 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae14a9a9-59d8-45c5-bff3-d01064363e59-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.369530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.369501 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.369720 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.369602 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.369799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.369782 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.369862 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.369851 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvsb5\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-kube-api-access-qvsb5\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.369920 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.369897 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.370028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.370016 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.424923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.424887 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc"] Nov 02 08:34:37.471120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.471083 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvsb5\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-kube-api-access-qvsb5\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.471355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.471143 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.471355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.471283 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.471483 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.471366 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.471483 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.471402 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.471483 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.471429 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae14a9a9-59d8-45c5-bff3-d01064363e59-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.471483 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.471472 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.471697 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.471537 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.471697 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.471655 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.472158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.472125 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-credential-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.472745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.472712 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-certs\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.472990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.472963 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-socket\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.474028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.474004 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae14a9a9-59d8-45c5-bff3-d01064363e59-istiod-ca-cert\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.474300 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.474279 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-data\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.475341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.475317 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-envoy\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.476051 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.476025 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-podinfo\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.526102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.526074 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-token\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.526266 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.526121 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvsb5\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-kube-api-access-qvsb5\") pod \"e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:37.645943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:37.645860 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:38.081934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:38.081902 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc"] Nov 02 08:34:38.082640 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:38.082572 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:34:38.085967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:38.082776 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:34:38.085967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:38.082888 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:34:38.855771 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:38.855728 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" event={"ID":"ae14a9a9-59d8-45c5-bff3-d01064363e59","Type":"ContainerStarted","Data":"7c5cf1fc5ed17591a21c210f7ec58036b7a135db544745b8da918837948fc088"} Nov 02 08:34:38.857049 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:38.857023 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" event={"ID":"ae14a9a9-59d8-45c5-bff3-d01064363e59","Type":"ContainerStarted","Data":"3049eab91d1c9dcc9c9f01088247d4c0d754a6db53a66ed767c3e6828c7afbd4"} Nov 02 08:34:38.969023 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:38.968967 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" podStartSLOduration=1.968951297 podStartE2EDuration="1.968951297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:38.967509994 +0000 UTC m=+1677.521014192" watchObservedRunningTime="2026-11-02 08:34:38.968951297 +0000 UTC m=+1677.522455494" Nov 02 08:34:39.646477 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:39.646443 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:39.648265 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:39.648237 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:34:39.648403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:39.648303 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 08:34:40.540281 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:34:40.540237 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:34:40.646780 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:40.646746 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:34:40.647321 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:40.646802 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 08:34:40.955058 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:40.955022 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:41.457889 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:41.457853 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:41.646870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:41.646837 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:34:41.647394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:41.646896 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 08:34:42.646800 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:42.646761 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:34:42.647000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:42.646860 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 08:34:43.646731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:43.646699 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:34:43.646911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:43.646760 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 08:34:44.004860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.004826 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc"] Nov 02 08:34:44.033887 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.033856 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.065151 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.065114 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc"] Nov 02 08:34:44.082555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.082522 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c17125-serving-cert\"" Nov 02 08:34:44.137297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.137255 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.137539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.137403 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.137539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.137481 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hrm\" (UniqueName: \"kubernetes.io/projected/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-kube-api-access-b4hrm\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.137687 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.137611 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.227810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.227773 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:44.239742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.239705 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.239908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.239827 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hrm\" (UniqueName: \"kubernetes.io/projected/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-kube-api-access-b4hrm\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.239908 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:34:44.239841 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-serving-cert: secret "maas-api-e2e-ipp-b-c17125-serving-cert" not found Nov 02 08:34:44.239908 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:34:44.239898 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls podName:bcb05d8b-3106-42b3-b257-eb1fb643fd0d nodeName:}" failed. No retries permitted until 2026-11-02 08:34:44.739880397 +0000 UTC m=+1683.293384585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls") pod "maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" (UID: "bcb05d8b-3106-42b3-b257-eb1fb643fd0d") : secret "maas-api-e2e-ipp-b-c17125-serving-cert" not found Nov 02 08:34:44.240046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.239914 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.240107 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.240093 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.240632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.240610 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.243137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.243109 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.309362 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.309268 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hrm\" (UniqueName: \"kubernetes.io/projected/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-kube-api-access-b4hrm\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.437069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.437024 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:44.597013 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.596912 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:44.652691 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.652661 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:44.653234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.653210 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:44.653909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.653888 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:34:44.749620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.749590 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.752213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.752168 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:44.948236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:44.948166 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:45.327913 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:34:45.327878 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb05d8b_3106_42b3_b257_eb1fb643fd0d.slice/crio-e88740171d50672bf6ab0567355443a1024ce921339d27e0387c5a86ae253e7b WatchSource:0}: Error finding container e88740171d50672bf6ab0567355443a1024ce921339d27e0387c5a86ae253e7b: Status 404 returned error can't find the container with id e88740171d50672bf6ab0567355443a1024ce921339d27e0387c5a86ae253e7b Nov 02 08:34:45.328262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:45.328099 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc"] Nov 02 08:34:45.329477 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:45.329458 6160 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:34:45.923877 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:45.923841 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" event={"ID":"bcb05d8b-3106-42b3-b257-eb1fb643fd0d","Type":"ContainerStarted","Data":"e88740171d50672bf6ab0567355443a1024ce921339d27e0387c5a86ae253e7b"} Nov 02 08:34:46.935455 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:46.935415 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" event={"ID":"bcb05d8b-3106-42b3-b257-eb1fb643fd0d","Type":"ContainerStarted","Data":"efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6"} Nov 02 08:34:46.935832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:46.935661 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:47.048524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:47.048456 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" podStartSLOduration=3.161440559 podStartE2EDuration="4.048439233s" totalImagesPullingTime="886.998674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:47.045240883 +0000 UTC m=+1685.598745080" watchObservedRunningTime="2026-11-02 08:34:47.048439233 +0000 UTC m=+1685.601943430" Nov 02 08:34:48.183436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:48.183400 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc"] Nov 02 08:34:48.438735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:48.438654 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:48.654413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:48.654376 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:48.818544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:48.818461 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:49.568110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:49.568069 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:50.911142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:50.911101 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:52.949994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:52.949966 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:34:53.184132 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:53.184087 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" podUID="702a68bf-93ae-43bb-92af-8061cc7ced66" containerName="payload-processing" containerID="cri-o://4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0" gracePeriod=25 Nov 02 08:34:53.467650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:53.467590 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:34:53.516632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:53.516601 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702a68bf-93ae-43bb-92af-8061cc7ced66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702a68bf-93ae-43bb-92af-8061cc7ced66-plugins-config-volume\") pod \"702a68bf-93ae-43bb-92af-8061cc7ced66\" (UID: \"702a68bf-93ae-43bb-92af-8061cc7ced66\") " Nov 02 08:34:53.516857 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:53.516841 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702a68bf-93ae-43bb-92af-8061cc7ced66-kube-api-access-45m9g\" (UniqueName: \"kubernetes.io/projected/702a68bf-93ae-43bb-92af-8061cc7ced66-kube-api-access-45m9g\") pod \"702a68bf-93ae-43bb-92af-8061cc7ced66\" (UID: \"702a68bf-93ae-43bb-92af-8061cc7ced66\") " Nov 02 08:34:53.516962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:53.516941 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702a68bf-93ae-43bb-92af-8061cc7ced66-plugins-config-volume" pod "702a68bf-93ae-43bb-92af-8061cc7ced66" (UID: "702a68bf-93ae-43bb-92af-8061cc7ced66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:34:53.517484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:53.517462 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702a68bf-93ae-43bb-92af-8061cc7ced66-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:53.519080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:53.519059 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702a68bf-93ae-43bb-92af-8061cc7ced66-kube-api-access-45m9g" pod "702a68bf-93ae-43bb-92af-8061cc7ced66" (UID: "702a68bf-93ae-43bb-92af-8061cc7ced66"). InnerVolumeSpecName "kube-api-access-45m9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:53.618547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:53.618512 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45m9g\" (UniqueName: \"kubernetes.io/projected/702a68bf-93ae-43bb-92af-8061cc7ced66-kube-api-access-45m9g\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:53.803028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:53.802949 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:54.010377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.010341 6160 generic.go:356] "Generic (PLEG): container finished" podID="702a68bf-93ae-43bb-92af-8061cc7ced66" containerID="4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0" exitCode=0 Nov 02 08:34:54.010834 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.010399 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" Nov 02 08:34:54.010834 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.010425 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" event={"ID":"702a68bf-93ae-43bb-92af-8061cc7ced66","Type":"ContainerDied","Data":"4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0"} Nov 02 08:34:54.010834 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.010522 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc" event={"ID":"702a68bf-93ae-43bb-92af-8061cc7ced66","Type":"ContainerDied","Data":"7fd58eab8c99139c24ad189879c104b1abc6425138e93f43deb8914599ea79f4"} Nov 02 08:34:54.010834 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.010547 6160 scope.go:122] "RemoveContainer" containerID="4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0" Nov 02 08:34:54.021410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.021389 6160 scope.go:122] "RemoveContainer" containerID="4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0" Nov 02 08:34:54.021676 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:34:54.021658 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0\": container with ID starting with 4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0 not found: ID does not exist" containerID="4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0" Nov 02 08:34:54.021723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.021686 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0"} err="failed to get container status \"4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0\": rpc error: code = NotFound desc = could not find container \"4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0\": container with ID starting with 4c0424228ebcf64bdc335673fefc95297ebebaff764daa681cd96dc5b324ece0 not found: ID does not exist" Nov 02 08:34:54.170796 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.170624 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc"] Nov 02 08:34:54.213696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.213660 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-56d6547467-tltlc"] Nov 02 08:34:54.521968 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.521882 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd"] Nov 02 08:34:54.522122 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:54.522099 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" containerID="cri-o://5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73" gracePeriod=30 Nov 02 08:34:55.404204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:55.404131 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:55.586592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:55.586558 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702a68bf-93ae-43bb-92af-8061cc7ced66" path="/var/lib/kubelet/pods/702a68bf-93ae-43bb-92af-8061cc7ced66/volumes" Nov 02 08:34:55.636797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:55.636762 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:55.804233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:55.804162 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:59.823434 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.823397 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:34:59.920362 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920332 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-certs\") pod \"c6917d45-23e7-49f5-b505-3536e900c6f8\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " Nov 02 08:34:59.920532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920480 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-envoy\") pod \"c6917d45-23e7-49f5-b505-3536e900c6f8\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " Nov 02 08:34:59.920532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920513 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-data\") pod \"c6917d45-23e7-49f5-b505-3536e900c6f8\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " Nov 02 08:34:59.920624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920548 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-socket\") pod \"c6917d45-23e7-49f5-b505-3536e900c6f8\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " Nov 02 08:34:59.920624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920586 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-token\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-token\") pod \"c6917d45-23e7-49f5-b505-3536e900c6f8\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " Nov 02 08:34:59.920736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920635 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-certs" pod "c6917d45-23e7-49f5-b505-3536e900c6f8" (UID: "c6917d45-23e7-49f5-b505-3536e900c6f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:59.920736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920674 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6917d45-23e7-49f5-b505-3536e900c6f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6917d45-23e7-49f5-b505-3536e900c6f8-istiod-ca-cert\") pod \"c6917d45-23e7-49f5-b505-3536e900c6f8\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " Nov 02 08:34:59.920736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920697 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-kube-api-access-9rb99\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-kube-api-access-9rb99\") pod \"c6917d45-23e7-49f5-b505-3536e900c6f8\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " Nov 02 08:34:59.920895 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920775 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-credential-socket\") pod \"c6917d45-23e7-49f5-b505-3536e900c6f8\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " Nov 02 08:34:59.920895 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920809 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-podinfo\") pod \"c6917d45-23e7-49f5-b505-3536e900c6f8\" (UID: \"c6917d45-23e7-49f5-b505-3536e900c6f8\") " Nov 02 08:34:59.920895 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920868 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-data" pod "c6917d45-23e7-49f5-b505-3536e900c6f8" (UID: "c6917d45-23e7-49f5-b505-3536e900c6f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:59.921092 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.920888 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-socket" pod "c6917d45-23e7-49f5-b505-3536e900c6f8" (UID: "c6917d45-23e7-49f5-b505-3536e900c6f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:59.921147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.921112 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-credential-socket" pod "c6917d45-23e7-49f5-b505-3536e900c6f8" (UID: "c6917d45-23e7-49f5-b505-3536e900c6f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:59.921147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.921120 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6917d45-23e7-49f5-b505-3536e900c6f8-istiod-ca-cert" pod "c6917d45-23e7-49f5-b505-3536e900c6f8" (UID: "c6917d45-23e7-49f5-b505-3536e900c6f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:34:59.921538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.921507 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:59.921538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.921527 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:59.921874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.921541 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6917d45-23e7-49f5-b505-3536e900c6f8-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:59.921874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.921555 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:59.921874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.921568 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:34:59.923106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.923083 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-kube-api-access-9rb99" pod "c6917d45-23e7-49f5-b505-3536e900c6f8" (UID: "c6917d45-23e7-49f5-b505-3536e900c6f8"). InnerVolumeSpecName "kube-api-access-9rb99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:34:59.923277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.923262 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-podinfo" pod "c6917d45-23e7-49f5-b505-3536e900c6f8" (UID: "c6917d45-23e7-49f5-b505-3536e900c6f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:34:59.923486 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.923472 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-envoy" pod "c6917d45-23e7-49f5-b505-3536e900c6f8" (UID: "c6917d45-23e7-49f5-b505-3536e900c6f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:34:59.923541 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:34:59.923520 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-token" pod "c6917d45-23e7-49f5-b505-3536e900c6f8" (UID: "c6917d45-23e7-49f5-b505-3536e900c6f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:00.022920 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.022884 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:00.022920 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.022916 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:00.023129 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.022932 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rb99\" (UniqueName: \"kubernetes.io/projected/c6917d45-23e7-49f5-b505-3536e900c6f8-kube-api-access-9rb99\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:00.023129 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.022949 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6917d45-23e7-49f5-b505-3536e900c6f8-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:00.057896 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.057799 6160 generic.go:356] "Generic (PLEG): container finished" podID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerID="5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73" exitCode=0 Nov 02 08:35:00.057896 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.057858 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" Nov 02 08:35:00.058125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.057892 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" event={"ID":"c6917d45-23e7-49f5-b505-3536e900c6f8","Type":"ContainerDied","Data":"5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73"} Nov 02 08:35:00.058125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.057950 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd" event={"ID":"c6917d45-23e7-49f5-b505-3536e900c6f8","Type":"ContainerDied","Data":"0019f3d40a4038e97ae5195ad27f8acc1bf32f069935a7c3c93841113d769191"} Nov 02 08:35:00.058125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.057973 6160 scope.go:122] "RemoveContainer" containerID="5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73" Nov 02 08:35:00.069988 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.069962 6160 scope.go:122] "RemoveContainer" containerID="5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73" Nov 02 08:35:00.070302 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:35:00.070283 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73\": container with ID starting with 5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73 not found: ID does not exist" containerID="5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73" Nov 02 08:35:00.070376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.070309 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73"} err="failed to get container status \"5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73\": rpc error: code = NotFound desc = could not find container \"5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73\": container with ID starting with 5922fb855158b93e5536e566361f28eafefa1880e14e43bc700ddf2c5b056b73 not found: ID does not exist" Nov 02 08:35:00.215648 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.215609 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd"] Nov 02 08:35:00.219564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.219533 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-nkchd"] Nov 02 08:35:00.697498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.697467 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:00.854640 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:00.854600 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:01.011238 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:01.011114 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:01.586369 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:01.586334 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" path="/var/lib/kubelet/pods/c6917d45-23e7-49f5-b505-3536e900c6f8/volumes" Nov 02 08:35:01.977588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:01.977551 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv"] Nov 02 08:35:01.978061 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:01.977760 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" containerID="cri-o://1c674d95b45afff467f63b17d99aa6546039817a1e5363d375c37d007bc63bb6" gracePeriod=30 Nov 02 08:35:06.045409 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:06.045378 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb"] Nov 02 08:35:06.091663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:06.091628 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cf78f1-hcnwb"] Nov 02 08:35:06.222880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:06.222842 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:07.132322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.132281 6160 generic.go:356] "Generic (PLEG): container finished" podID="702d810a-cd06-4807-9470-17613fda7434" containerID="1c674d95b45afff467f63b17d99aa6546039817a1e5363d375c37d007bc63bb6" exitCode=0 Nov 02 08:35:07.132710 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.132352 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" event={"ID":"702d810a-cd06-4807-9470-17613fda7434","Type":"ContainerDied","Data":"1c674d95b45afff467f63b17d99aa6546039817a1e5363d375c37d007bc63bb6"} Nov 02 08:35:07.252847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.252781 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:35:07.325420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325386 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-data\") pod \"702d810a-cd06-4807-9470-17613fda7434\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " Nov 02 08:35:07.325604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325444 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-credential-socket\") pod \"702d810a-cd06-4807-9470-17613fda7434\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " Nov 02 08:35:07.325604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325479 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-istio-token\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-istio-token\") pod \"702d810a-cd06-4807-9470-17613fda7434\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " Nov 02 08:35:07.325604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325505 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702d810a-cd06-4807-9470-17613fda7434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702d810a-cd06-4807-9470-17613fda7434-istiod-ca-cert\") pod \"702d810a-cd06-4807-9470-17613fda7434\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " Nov 02 08:35:07.325799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325637 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/702d810a-cd06-4807-9470-17613fda7434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702d810a-cd06-4807-9470-17613fda7434-istio-podinfo\") pod \"702d810a-cd06-4807-9470-17613fda7434\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " Nov 02 08:35:07.325799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325691 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-kube-api-access-2qr66\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-kube-api-access-2qr66\") pod \"702d810a-cd06-4807-9470-17613fda7434\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " Nov 02 08:35:07.325799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325727 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-data" pod "702d810a-cd06-4807-9470-17613fda7434" (UID: "702d810a-cd06-4807-9470-17613fda7434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:07.325987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325796 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-credential-socket" pod "702d810a-cd06-4807-9470-17613fda7434" (UID: "702d810a-cd06-4807-9470-17613fda7434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:07.325987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325837 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-envoy\") pod \"702d810a-cd06-4807-9470-17613fda7434\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " Nov 02 08:35:07.325987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325870 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-socket\") pod \"702d810a-cd06-4807-9470-17613fda7434\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " Nov 02 08:35:07.325987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325942 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-certs\") pod \"702d810a-cd06-4807-9470-17613fda7434\" (UID: \"702d810a-cd06-4807-9470-17613fda7434\") " Nov 02 08:35:07.325987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.325981 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702d810a-cd06-4807-9470-17613fda7434-istiod-ca-cert" pod "702d810a-cd06-4807-9470-17613fda7434" (UID: "702d810a-cd06-4807-9470-17613fda7434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:07.326286 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.326270 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-socket" pod "702d810a-cd06-4807-9470-17613fda7434" (UID: "702d810a-cd06-4807-9470-17613fda7434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:07.326482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.326374 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-certs" pod "702d810a-cd06-4807-9470-17613fda7434" (UID: "702d810a-cd06-4807-9470-17613fda7434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:07.326662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.326643 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:07.326728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.326667 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:07.326728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.326681 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702d810a-cd06-4807-9470-17613fda7434-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:07.326728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.326697 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:07.326728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.326710 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:07.328010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.327988 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/702d810a-cd06-4807-9470-17613fda7434-istio-podinfo" pod "702d810a-cd06-4807-9470-17613fda7434" (UID: "702d810a-cd06-4807-9470-17613fda7434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:35:07.328343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.328321 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-istio-token" pod "702d810a-cd06-4807-9470-17613fda7434" (UID: "702d810a-cd06-4807-9470-17613fda7434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:07.328580 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.328561 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-kube-api-access-2qr66" pod "702d810a-cd06-4807-9470-17613fda7434" (UID: "702d810a-cd06-4807-9470-17613fda7434"). InnerVolumeSpecName "kube-api-access-2qr66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:07.328629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.328561 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-envoy" pod "702d810a-cd06-4807-9470-17613fda7434" (UID: "702d810a-cd06-4807-9470-17613fda7434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:07.428634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.428601 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:07.428634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.428632 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702d810a-cd06-4807-9470-17613fda7434-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:07.428634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.428642 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qr66\" (UniqueName: \"kubernetes.io/projected/702d810a-cd06-4807-9470-17613fda7434-kube-api-access-2qr66\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:07.428863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.428653 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702d810a-cd06-4807-9470-17613fda7434-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:07.585357 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:07.585279 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c62352-c357-4856-8f73-4183fbba0c89" path="/var/lib/kubelet/pods/49c62352-c357-4856-8f73-4183fbba0c89/volumes" Nov 02 08:35:08.142150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:08.142122 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" Nov 02 08:35:08.142643 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:08.142162 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv" event={"ID":"702d810a-cd06-4807-9470-17613fda7434","Type":"ContainerDied","Data":"8c1397097083e2ecd853dba96be0d93eb351e2d01b7b249693188a69233a3f1f"} Nov 02 08:35:08.142643 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:08.142264 6160 scope.go:122] "RemoveContainer" containerID="1c674d95b45afff467f63b17d99aa6546039817a1e5363d375c37d007bc63bb6" Nov 02 08:35:08.324403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:08.324368 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv"] Nov 02 08:35:08.328078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:08.328058 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-pmmbv"] Nov 02 08:35:09.585192 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:09.585152 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702d810a-cd06-4807-9470-17613fda7434" path="/var/lib/kubelet/pods/702d810a-cd06-4807-9470-17613fda7434/volumes" Nov 02 08:35:10.522349 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:35:10.522315 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:35:11.378045 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:11.378008 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:16.148946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:16.148907 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:19.814637 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:19.814598 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:21.255693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:21.255649 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:22.168896 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:22.168858 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:26.056995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:26.056952 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:29.451595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:29.451559 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:30.086558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.086519 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb"] Nov 02 08:35:30.086773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.086719 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" podUID="52882ba1-9a64-446b-bf7b-5324e1d1a545" containerName="maas-api" containerID="cri-o://5f8edae10915dbbd3cb6ab16d7c1fd7d8d88f8038a330f0b0f07c3dd8b740610" gracePeriod=30 Nov 02 08:35:30.135130 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.135096 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:30.328236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.328200 6160 generic.go:356] "Generic (PLEG): container finished" podID="52882ba1-9a64-446b-bf7b-5324e1d1a545" containerID="5f8edae10915dbbd3cb6ab16d7c1fd7d8d88f8038a330f0b0f07c3dd8b740610" exitCode=0 Nov 02 08:35:30.328438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.328419 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" event={"ID":"52882ba1-9a64-446b-bf7b-5324e1d1a545","Type":"ContainerDied","Data":"5f8edae10915dbbd3cb6ab16d7c1fd7d8d88f8038a330f0b0f07c3dd8b740610"} Nov 02 08:35:30.341496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.341344 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:30.403104 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.403071 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:35:30.497715 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.497677 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52882ba1-9a64-446b-bf7b-5324e1d1a545-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52882ba1-9a64-446b-bf7b-5324e1d1a545-openshift-service-ca\") pod \"52882ba1-9a64-446b-bf7b-5324e1d1a545\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " Nov 02 08:35:30.498407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.498383 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-metrics-certs\") pod \"52882ba1-9a64-446b-bf7b-5324e1d1a545\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " Nov 02 08:35:30.499112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.498648 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52882ba1-9a64-446b-bf7b-5324e1d1a545-openshift-service-ca" pod "52882ba1-9a64-446b-bf7b-5324e1d1a545" (UID: "52882ba1-9a64-446b-bf7b-5324e1d1a545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:30.499603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.499055 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-maas-api-tls\") pod \"52882ba1-9a64-446b-bf7b-5324e1d1a545\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " Nov 02 08:35:30.499990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.499973 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52882ba1-9a64-446b-bf7b-5324e1d1a545-kube-api-access-m7877\" (UniqueName: \"kubernetes.io/projected/52882ba1-9a64-446b-bf7b-5324e1d1a545-kube-api-access-m7877\") pod \"52882ba1-9a64-446b-bf7b-5324e1d1a545\" (UID: \"52882ba1-9a64-446b-bf7b-5324e1d1a545\") " Nov 02 08:35:30.501374 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.501355 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52882ba1-9a64-446b-bf7b-5324e1d1a545-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:30.501608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.501583 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-metrics-certs" pod "52882ba1-9a64-446b-bf7b-5324e1d1a545" (UID: "52882ba1-9a64-446b-bf7b-5324e1d1a545"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:30.517544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.517485 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-maas-api-tls" pod "52882ba1-9a64-446b-bf7b-5324e1d1a545" (UID: "52882ba1-9a64-446b-bf7b-5324e1d1a545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:30.524270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.524240 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52882ba1-9a64-446b-bf7b-5324e1d1a545-kube-api-access-m7877" pod "52882ba1-9a64-446b-bf7b-5324e1d1a545" (UID: "52882ba1-9a64-446b-bf7b-5324e1d1a545"). InnerVolumeSpecName "kube-api-access-m7877". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:30.602647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.602565 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:30.602647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.602593 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52882ba1-9a64-446b-bf7b-5324e1d1a545-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:30.602647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:30.602603 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7877\" (UniqueName: \"kubernetes.io/projected/52882ba1-9a64-446b-bf7b-5324e1d1a545-kube-api-access-m7877\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:31.337484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:31.337452 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" Nov 02 08:35:31.337484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:31.337477 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb" event={"ID":"52882ba1-9a64-446b-bf7b-5324e1d1a545","Type":"ContainerDied","Data":"61e03e7e6b4b458474d12600136a5bb6bf165987ac60c388b73abefd3865901c"} Nov 02 08:35:31.337736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:31.337546 6160 scope.go:122] "RemoveContainer" containerID="5f8edae10915dbbd3cb6ab16d7c1fd7d8d88f8038a330f0b0f07c3dd8b740610" Nov 02 08:35:31.492422 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:31.492384 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb"] Nov 02 08:35:31.496159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:31.496137 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5bb4f-9f7b96bcd-xjdlb"] Nov 02 08:35:31.586820 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:31.586788 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52882ba1-9a64-446b-bf7b-5324e1d1a545" path="/var/lib/kubelet/pods/52882ba1-9a64-446b-bf7b-5324e1d1a545/volumes" Nov 02 08:35:31.810672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:31.810639 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:32.052094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:32.052059 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:32.208727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:32.208693 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:32.699896 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:32.699858 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:36.377711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:36.377668 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:36.606153 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:36.606120 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:37.264367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:37.264336 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:38.722727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:38.722694 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:38.881771 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:38.881735 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:39.049482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:39.049383 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:39.217236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:39.213672 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:40.523018 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:35:40.522984 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:35:42.425949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:42.425908 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:42.790823 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:42.790716 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:49.729254 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:49.729213 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:51.215837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:51.215797 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:51.507215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:51.507124 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb"] Nov 02 08:35:51.622659 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:51.622622 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52"] Nov 02 08:35:51.941679 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:51.941647 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:52.157103 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:52.157056 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:54.506958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:54.506924 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr"] Nov 02 08:35:54.507385 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:54.507126 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" containerID="cri-o://ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94" gracePeriod=30 Nov 02 08:35:55.182731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:55.182694 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:56.508531 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:56.508487 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" podUID="dd3f86dd-fa7e-459a-a49d-c2f710f1bead" containerName="payload-processing" containerID="cri-o://72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa" gracePeriod=25 Nov 02 08:35:56.625105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:56.625068 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" podUID="1fbfc5e6-88b9-4576-b121-0e4ef0780368" containerName="payload-pre-processing" containerID="cri-o://1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c" gracePeriod=25 Nov 02 08:35:56.806352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:56.806320 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:35:56.925315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:56.925289 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:35:56.988418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:56.988388 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-kube-api-access-mgbzp\" (UniqueName: \"kubernetes.io/projected/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-kube-api-access-mgbzp\") pod \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\" (UID: \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\") " Nov 02 08:35:56.988633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:56.988618 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-plugins-config-volume\") pod \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\" (UID: \"dd3f86dd-fa7e-459a-a49d-c2f710f1bead\") " Nov 02 08:35:56.988966 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:56.988942 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-plugins-config-volume" pod "dd3f86dd-fa7e-459a-a49d-c2f710f1bead" (UID: "dd3f86dd-fa7e-459a-a49d-c2f710f1bead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:56.989091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:56.989073 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:56.990708 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:56.990686 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-kube-api-access-mgbzp" pod "dd3f86dd-fa7e-459a-a49d-c2f710f1bead" (UID: "dd3f86dd-fa7e-459a-a49d-c2f710f1bead"). InnerVolumeSpecName "kube-api-access-mgbzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:57.089874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.089789 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fbfc5e6-88b9-4576-b121-0e4ef0780368-kube-api-access-29xfj\" (UniqueName: \"kubernetes.io/projected/1fbfc5e6-88b9-4576-b121-0e4ef0780368-kube-api-access-29xfj\") pod \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\" (UID: \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\") " Nov 02 08:35:57.090026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.089890 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fbfc5e6-88b9-4576-b121-0e4ef0780368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fbfc5e6-88b9-4576-b121-0e4ef0780368-plugins-config-volume\") pod \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\" (UID: \"1fbfc5e6-88b9-4576-b121-0e4ef0780368\") " Nov 02 08:35:57.090384 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.090353 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fbfc5e6-88b9-4576-b121-0e4ef0780368-plugins-config-volume" pod "1fbfc5e6-88b9-4576-b121-0e4ef0780368" (UID: "1fbfc5e6-88b9-4576-b121-0e4ef0780368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:57.090617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.090568 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fbfc5e6-88b9-4576-b121-0e4ef0780368-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:57.090617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.090596 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgbzp\" (UniqueName: \"kubernetes.io/projected/dd3f86dd-fa7e-459a-a49d-c2f710f1bead-kube-api-access-mgbzp\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:57.092337 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.092313 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbfc5e6-88b9-4576-b121-0e4ef0780368-kube-api-access-29xfj" pod "1fbfc5e6-88b9-4576-b121-0e4ef0780368" (UID: "1fbfc5e6-88b9-4576-b121-0e4ef0780368"). InnerVolumeSpecName "kube-api-access-29xfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:57.191599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.191560 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29xfj\" (UniqueName: \"kubernetes.io/projected/1fbfc5e6-88b9-4576-b121-0e4ef0780368-kube-api-access-29xfj\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:57.543236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.543165 6160 generic.go:356] "Generic (PLEG): container finished" podID="1fbfc5e6-88b9-4576-b121-0e4ef0780368" containerID="1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c" exitCode=0 Nov 02 08:35:57.543665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.543272 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" Nov 02 08:35:57.543665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.543303 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" event={"ID":"1fbfc5e6-88b9-4576-b121-0e4ef0780368","Type":"ContainerDied","Data":"1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c"} Nov 02 08:35:57.543665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.543545 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52" event={"ID":"1fbfc5e6-88b9-4576-b121-0e4ef0780368","Type":"ContainerDied","Data":"dd97a4334e2faf0115119b632aad088204705e3e88c2b8661f1dd05001c588b9"} Nov 02 08:35:57.543665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.543590 6160 scope.go:122] "RemoveContainer" containerID="1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c" Nov 02 08:35:57.545746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.545715 6160 generic.go:356] "Generic (PLEG): container finished" podID="dd3f86dd-fa7e-459a-a49d-c2f710f1bead" containerID="72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa" exitCode=0 Nov 02 08:35:57.545886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.545777 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" Nov 02 08:35:57.545886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.545812 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" event={"ID":"dd3f86dd-fa7e-459a-a49d-c2f710f1bead","Type":"ContainerDied","Data":"72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa"} Nov 02 08:35:57.545886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.545855 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb" event={"ID":"dd3f86dd-fa7e-459a-a49d-c2f710f1bead","Type":"ContainerDied","Data":"cc0f754df3a40f58d96e44674206c976676ce1b2b446a016691a40c600e02cc4"} Nov 02 08:35:57.555612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.555585 6160 scope.go:122] "RemoveContainer" containerID="1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c" Nov 02 08:35:57.555939 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:35:57.555919 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c\": container with ID starting with 1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c not found: ID does not exist" containerID="1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c" Nov 02 08:35:57.556045 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.555946 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c"} err="failed to get container status \"1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c\": rpc error: code = NotFound desc = could not find container \"1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c\": container with ID starting with 1e0a88b22bbb7b3b7676049252c9e31533a16a226c4d13ab587df06d54b2709c not found: ID does not exist" Nov 02 08:35:57.556045 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.555970 6160 scope.go:122] "RemoveContainer" containerID="72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa" Nov 02 08:35:57.566040 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.566024 6160 scope.go:122] "RemoveContainer" containerID="72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa" Nov 02 08:35:57.566378 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:35:57.566347 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa\": container with ID starting with 72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa not found: ID does not exist" containerID="72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa" Nov 02 08:35:57.566467 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.566378 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa"} err="failed to get container status \"72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa\": rpc error: code = NotFound desc = could not find container \"72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa\": container with ID starting with 72fc5529f0602f7eee60ebf6d8d34772f487fea0b04e82564b218d6bf7fbafaa not found: ID does not exist" Nov 02 08:35:57.698773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.698737 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb"] Nov 02 08:35:57.744250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.744205 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a25202-7f74f7945-bp6zb"] Nov 02 08:35:57.805507 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.805402 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52"] Nov 02 08:35:57.811707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:57.811679 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a25202-8c9b8c975-6dh52"] Nov 02 08:35:59.586504 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.586469 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fbfc5e6-88b9-4576-b121-0e4ef0780368" path="/var/lib/kubelet/pods/1fbfc5e6-88b9-4576-b121-0e4ef0780368/volumes" Nov 02 08:35:59.586935 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.586921 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3f86dd-fa7e-459a-a49d-c2f710f1bead" path="/var/lib/kubelet/pods/dd3f86dd-fa7e-459a-a49d-c2f710f1bead/volumes" Nov 02 08:35:59.796562 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.796483 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:35:59.831095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831062 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-data\") pod \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " Nov 02 08:35:59.831296 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831105 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istiod-ca-cert\") pod \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " Nov 02 08:35:59.831296 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831153 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-kube-api-access-tsbl2\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-kube-api-access-tsbl2\") pod \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " Nov 02 08:35:59.831296 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831262 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-credential-socket\") pod \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " Nov 02 08:35:59.831470 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831333 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-data" pod "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" (UID: "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.831470 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831361 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-envoy\") pod \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " Nov 02 08:35:59.831574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831492 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-socket\") pod \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " Nov 02 08:35:59.831574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831538 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istiod-ca-cert" pod "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" (UID: "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:59.831574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831555 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-podinfo\") pod \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " Nov 02 08:35:59.831574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831556 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-credential-socket" pod "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" (UID: "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.831782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831601 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-token\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-token\") pod \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " Nov 02 08:35:59.831782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831630 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-certs\") pod \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\" (UID: \"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0\") " Nov 02 08:35:59.831865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831799 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-socket" pod "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" (UID: "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.832012 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.831986 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-certs" pod "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" (UID: "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.832493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.832421 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:59.832598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.832500 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:59.832598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.832514 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:59.832598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.832529 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:59.832598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.832543 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:59.834116 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.834091 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-envoy" pod "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" (UID: "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:59.834220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.834113 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-kube-api-access-tsbl2" pod "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" (UID: "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0"). InnerVolumeSpecName "kube-api-access-tsbl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:59.834413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.834392 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-podinfo" pod "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" (UID: "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:35:59.834473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.834397 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-token" pod "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" (UID: "342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:59.933759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.933716 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:59.933759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.933758 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:59.934008 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.933773 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:35:59.934008 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:35:59.933788 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsbl2\" (UniqueName: \"kubernetes.io/projected/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0-kube-api-access-tsbl2\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:00.571341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.571306 6160 generic.go:356] "Generic (PLEG): container finished" podID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerID="ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94" exitCode=0 Nov 02 08:36:00.571568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.571359 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" Nov 02 08:36:00.571568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.571390 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" event={"ID":"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0","Type":"ContainerDied","Data":"ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94"} Nov 02 08:36:00.571568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.571476 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr" event={"ID":"342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0","Type":"ContainerDied","Data":"cfd0119a8ed3ee12100265e9f123276a1551187a079413666bba85a740a5e0b5"} Nov 02 08:36:00.571568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.571505 6160 scope.go:122] "RemoveContainer" containerID="ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94" Nov 02 08:36:00.584165 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.584144 6160 scope.go:122] "RemoveContainer" containerID="ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94" Nov 02 08:36:00.584492 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:36:00.584472 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94\": container with ID starting with ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94 not found: ID does not exist" containerID="ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94" Nov 02 08:36:00.584546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.584500 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94"} err="failed to get container status \"ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94\": rpc error: code = NotFound desc = could not find container \"ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94\": container with ID starting with ac18d549876a64374b79e4d38ff7be449759d126d3d73706d1465fee55693a94 not found: ID does not exist" Nov 02 08:36:00.760898 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.760858 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:00.766280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.766250 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr"] Nov 02 08:36:00.812168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:00.812137 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-tkmcr"] Nov 02 08:36:01.587571 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:01.587533 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" path="/var/lib/kubelet/pods/342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0/volumes" Nov 02 08:36:02.436860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:02.436819 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:04.564267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:04.564227 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:05.124379 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.124332 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:05.348754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.348715 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6"] Nov 02 08:36:05.349527 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349504 6160 cpu_manager.go:414] "containerMap: removing container" podUID="52882ba1-9a64-446b-bf7b-5324e1d1a545" containerName="maas-api" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349533 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52882ba1-9a64-446b-bf7b-5324e1d1a545" containerName="maas-api" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349555 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349561 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349567 6160 cpu_manager.go:414] "containerMap: removing container" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349573 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349590 6160 cpu_manager.go:414] "containerMap: removing container" podUID="1fbfc5e6-88b9-4576-b121-0e4ef0780368" containerName="payload-pre-processing" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349594 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fbfc5e6-88b9-4576-b121-0e4ef0780368" containerName="payload-pre-processing" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349620 6160 cpu_manager.go:414] "containerMap: removing container" podUID="702a68bf-93ae-43bb-92af-8061cc7ced66" containerName="payload-processing" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349629 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702a68bf-93ae-43bb-92af-8061cc7ced66" containerName="payload-processing" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349650 6160 cpu_manager.go:414] "containerMap: removing container" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349654 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349666 6160 cpu_manager.go:414] "containerMap: removing container" podUID="dd3f86dd-fa7e-459a-a49d-c2f710f1bead" containerName="payload-processing" Nov 02 08:36:05.349677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349670 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3f86dd-fa7e-459a-a49d-c2f710f1bead" containerName="payload-processing" Nov 02 08:36:05.350244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349778 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="702a68bf-93ae-43bb-92af-8061cc7ced66" containerName="payload-processing" Nov 02 08:36:05.350244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349793 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="702d810a-cd06-4807-9470-17613fda7434" containerName="istio-proxy" Nov 02 08:36:05.350244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349798 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3f86dd-fa7e-459a-a49d-c2f710f1bead" containerName="payload-processing" Nov 02 08:36:05.350244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349806 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="52882ba1-9a64-446b-bf7b-5324e1d1a545" containerName="maas-api" Nov 02 08:36:05.350244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349811 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6917d45-23e7-49f5-b505-3536e900c6f8" containerName="istio-proxy" Nov 02 08:36:05.350244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349816 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="342ae15a-1d5c-4860-9eaa-0fc9b2cb27c0" containerName="istio-proxy" Nov 02 08:36:05.350244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.349824 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fbfc5e6-88b9-4576-b121-0e4ef0780368" containerName="payload-pre-processing" Nov 02 08:36:05.353851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.353830 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.407557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.407472 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6"] Nov 02 08:36:05.414528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.414499 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.414693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.414543 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.414693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.414611 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbm4h\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-kube-api-access-mbm4h\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.414693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.414675 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.414833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.414747 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.414906 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.414882 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.415014 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.414994 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.415060 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.415037 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.415112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.415099 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d527bf-dd46-4099-a154-789eb5d38ca1-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.448608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.448574 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b9eea3-openshift-default-dockercfg-lt5rj\"" Nov 02 08:36:05.516375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516345 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.516589 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516390 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.516589 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516436 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d527bf-dd46-4099-a154-789eb5d38ca1-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.516589 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516504 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.516589 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516540 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.516589 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516584 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbm4h\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-kube-api-access-mbm4h\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.516884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516616 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.516884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516672 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.516884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516760 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.516884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.516798 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.517094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.517012 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.517094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.517052 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.517229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.517211 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.517276 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.517251 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d527bf-dd46-4099-a154-789eb5d38ca1-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.519281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.519260 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.519424 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.519407 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.590167 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.590132 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.597511 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.597477 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbm4h\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-kube-api-access-mbm4h\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:05.672488 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:05.672452 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:06.103130 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.103096 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6"] Nov 02 08:36:06.103477 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:36:06.103453 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d527bf_dd46_4099_a154_789eb5d38ca1.slice/crio-063329750c33f04cf5e67605e72daed498ab92765cc721324b02a78e50f80fd6 WatchSource:0}: Error finding container 063329750c33f04cf5e67605e72daed498ab92765cc721324b02a78e50f80fd6: Status 404 returned error can't find the container with id 063329750c33f04cf5e67605e72daed498ab92765cc721324b02a78e50f80fd6 Nov 02 08:36:06.106155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.106123 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:36:06.106249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.106231 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:36:06.106288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.106265 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:36:06.193031 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.192999 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx"] Nov 02 08:36:06.197424 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.197400 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:06.226382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.226353 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmsj\" (UniqueName: \"kubernetes.io/projected/c9a8db27-6990-4903-b0fe-73f712e5a59a-kube-api-access-dcmsj\") pod \"payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx\" (UID: \"c9a8db27-6990-4903-b0fe-73f712e5a59a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:06.226584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.226560 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a8db27-6990-4903-b0fe-73f712e5a59a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx\" (UID: \"c9a8db27-6990-4903-b0fe-73f712e5a59a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:06.250773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.247840 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-69a9bb-dockercfg-xpwvc\"" Nov 02 08:36:06.250773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.247959 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-69a9bb\"" Nov 02 08:36:06.253462 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.253327 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx"] Nov 02 08:36:06.328742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.328693 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmsj\" (UniqueName: \"kubernetes.io/projected/c9a8db27-6990-4903-b0fe-73f712e5a59a-kube-api-access-dcmsj\") pod \"payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx\" (UID: \"c9a8db27-6990-4903-b0fe-73f712e5a59a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:06.328905 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.328887 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a8db27-6990-4903-b0fe-73f712e5a59a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx\" (UID: \"c9a8db27-6990-4903-b0fe-73f712e5a59a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:06.329848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.329820 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a8db27-6990-4903-b0fe-73f712e5a59a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx\" (UID: \"c9a8db27-6990-4903-b0fe-73f712e5a59a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:06.381839 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.381747 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmsj\" (UniqueName: \"kubernetes.io/projected/c9a8db27-6990-4903-b0fe-73f712e5a59a-kube-api-access-dcmsj\") pod \"payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx\" (UID: \"c9a8db27-6990-4903-b0fe-73f712e5a59a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:06.515801 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.515758 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:06.658261 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.658130 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" event={"ID":"a9d527bf-dd46-4099-a154-789eb5d38ca1","Type":"ContainerStarted","Data":"d86ae705362ed3492d4d3bade3950151c23ad2bbde62ce8424163e72cdde4b31"} Nov 02 08:36:06.658261 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.658228 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" event={"ID":"a9d527bf-dd46-4099-a154-789eb5d38ca1","Type":"ContainerStarted","Data":"063329750c33f04cf5e67605e72daed498ab92765cc721324b02a78e50f80fd6"} Nov 02 08:36:06.673502 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.673469 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:06.771714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.771656 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" podStartSLOduration=1.77164085 podStartE2EDuration="1.77164085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:06.767020238 +0000 UTC m=+1765.320524462" watchObservedRunningTime="2026-11-02 08:36:06.77164085 +0000 UTC m=+1765.325145047" Nov 02 08:36:06.882385 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.882341 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh"] Nov 02 08:36:06.899202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.895690 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:06.901942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.901272 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx"] Nov 02 08:36:06.904100 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:36:06.904067 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a8db27_6990_4903_b0fe_73f712e5a59a.slice/crio-8514e6e0e6f974c868355f4cebda4ed19b152994f16653757ee047653ad0a39d WatchSource:0}: Error finding container 8514e6e0e6f974c868355f4cebda4ed19b152994f16653757ee047653ad0a39d: Status 404 returned error can't find the container with id 8514e6e0e6f974c868355f4cebda4ed19b152994f16653757ee047653ad0a39d Nov 02 08:36:06.939295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.937961 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh\" (UID: \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:06.939295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.938294 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9wmk\" (UniqueName: \"kubernetes.io/projected/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-kube-api-access-x9wmk\") pod \"payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh\" (UID: \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:06.957818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:06.957781 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh"] Nov 02 08:36:07.039919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.039885 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh\" (UID: \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:07.040091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.039984 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9wmk\" (UniqueName: \"kubernetes.io/projected/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-kube-api-access-x9wmk\") pod \"payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh\" (UID: \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:07.040931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.040910 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh\" (UID: \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:07.093875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.093836 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9wmk\" (UniqueName: \"kubernetes.io/projected/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-kube-api-access-x9wmk\") pod \"payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh\" (UID: \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:07.259279 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.259159 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:07.664816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.664773 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh"] Nov 02 08:36:07.665610 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:36:07.665532 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab5677e_be56_4ada_a457_7ee7d9ad4cda.slice/crio-6a2f78a315d6a647484438353d80cb6fc5aefed4dcd7a7bd8d35196e3a7d493d WatchSource:0}: Error finding container 6a2f78a315d6a647484438353d80cb6fc5aefed4dcd7a7bd8d35196e3a7d493d: Status 404 returned error can't find the container with id 6a2f78a315d6a647484438353d80cb6fc5aefed4dcd7a7bd8d35196e3a7d493d Nov 02 08:36:07.670316 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.670259 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" event={"ID":"c9a8db27-6990-4903-b0fe-73f712e5a59a","Type":"ContainerStarted","Data":"83c1f7c6c78a2be7152acdab95246fd67b8dde8110f011062ca0843da5444e18"} Nov 02 08:36:07.670424 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.670332 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" event={"ID":"c9a8db27-6990-4903-b0fe-73f712e5a59a","Type":"ContainerStarted","Data":"8514e6e0e6f974c868355f4cebda4ed19b152994f16653757ee047653ad0a39d"} Nov 02 08:36:07.670770 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.670740 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:07.674493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.674472 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 08:36:07.674583 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.674512 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 08:36:07.789860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:07.789814 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" podStartSLOduration=1.7897998080000002 podStartE2EDuration="1.789799808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:07.787026306 +0000 UTC m=+1766.340530504" watchObservedRunningTime="2026-11-02 08:36:07.789799808 +0000 UTC m=+1766.343304005" Nov 02 08:36:08.045863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:08.045804 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:08.673934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:08.673901 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 08:36:08.674427 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:08.673961 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 08:36:08.686551 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:08.686500 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" event={"ID":"7ab5677e-be56-4ada-a457-7ee7d9ad4cda","Type":"ContainerStarted","Data":"34a55dd861414af2675a2173746962ced44b6df7b71c8ddfc8bfdf5988c28323"} Nov 02 08:36:08.686761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:08.686566 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" event={"ID":"7ab5677e-be56-4ada-a457-7ee7d9ad4cda","Type":"ContainerStarted","Data":"6a2f78a315d6a647484438353d80cb6fc5aefed4dcd7a7bd8d35196e3a7d493d"} Nov 02 08:36:08.686761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:08.686594 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:08.802113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:08.802048 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" podStartSLOduration=2.802030613 podStartE2EDuration="2.802030613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:08.79614891 +0000 UTC m=+1767.349653088" watchObservedRunningTime="2026-11-02 08:36:08.802030613 +0000 UTC m=+1767.355534810" Nov 02 08:36:09.438090 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:09.437262 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf"] Nov 02 08:36:09.673277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:09.673246 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 08:36:09.673456 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:09.673301 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 08:36:09.833011 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:09.832910 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:09.985743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:09.985708 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:10.543827 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:36:10.543773 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:36:10.673163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:10.673131 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 08:36:10.673382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:10.673209 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 08:36:11.673778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:11.673742 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 08:36:11.674212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:11.673801 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 08:36:12.678314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:12.678284 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:12.678799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:12.678674 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:12.679476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:12.679454 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:36:14.036230 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.036153 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:14.438737 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.438686 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" podUID="52606d0f-465b-487e-85ea-d6a7f45256ba" containerName="payload-processing" containerID="cri-o://0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0" gracePeriod=25 Nov 02 08:36:14.744778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.744745 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:36:14.759049 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.759014 6160 generic.go:356] "Generic (PLEG): container finished" podID="52606d0f-465b-487e-85ea-d6a7f45256ba" containerID="0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0" exitCode=0 Nov 02 08:36:14.759220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.759057 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" Nov 02 08:36:14.759220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.759100 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" event={"ID":"52606d0f-465b-487e-85ea-d6a7f45256ba","Type":"ContainerDied","Data":"0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0"} Nov 02 08:36:14.759220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.759154 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf" event={"ID":"52606d0f-465b-487e-85ea-d6a7f45256ba","Type":"ContainerDied","Data":"e7d66c3dd719f761019670cbf5775e86cb5606352490ef26e2cc9e3eb8c16c19"} Nov 02 08:36:14.759220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.759204 6160 scope.go:122] "RemoveContainer" containerID="0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0" Nov 02 08:36:14.770096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.770079 6160 scope.go:122] "RemoveContainer" containerID="0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0" Nov 02 08:36:14.770351 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:36:14.770334 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0\": container with ID starting with 0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0 not found: ID does not exist" containerID="0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0" Nov 02 08:36:14.770403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.770359 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0"} err="failed to get container status \"0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0\": rpc error: code = NotFound desc = could not find container \"0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0\": container with ID starting with 0383aee93dac5227d0446e242888f2c9ac2c7ef498d338ca92366768f1b60fa0 not found: ID does not exist" Nov 02 08:36:14.885544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.883399 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52606d0f-465b-487e-85ea-d6a7f45256ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606d0f-465b-487e-85ea-d6a7f45256ba-plugins-config-volume\") pod \"52606d0f-465b-487e-85ea-d6a7f45256ba\" (UID: \"52606d0f-465b-487e-85ea-d6a7f45256ba\") " Nov 02 08:36:14.885544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.883545 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52606d0f-465b-487e-85ea-d6a7f45256ba-kube-api-access-w2q9g\" (UniqueName: \"kubernetes.io/projected/52606d0f-465b-487e-85ea-d6a7f45256ba-kube-api-access-w2q9g\") pod \"52606d0f-465b-487e-85ea-d6a7f45256ba\" (UID: \"52606d0f-465b-487e-85ea-d6a7f45256ba\") " Nov 02 08:36:14.885544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.884367 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52606d0f-465b-487e-85ea-d6a7f45256ba-plugins-config-volume" pod "52606d0f-465b-487e-85ea-d6a7f45256ba" (UID: "52606d0f-465b-487e-85ea-d6a7f45256ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:14.885544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:14.884716 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606d0f-465b-487e-85ea-d6a7f45256ba-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:15.001765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:15.001690 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52606d0f-465b-487e-85ea-d6a7f45256ba-kube-api-access-w2q9g" pod "52606d0f-465b-487e-85ea-d6a7f45256ba" (UID: "52606d0f-465b-487e-85ea-d6a7f45256ba"). InnerVolumeSpecName "kube-api-access-w2q9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:15.087719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:15.087686 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2q9g\" (UniqueName: \"kubernetes.io/projected/52606d0f-465b-487e-85ea-d6a7f45256ba-kube-api-access-w2q9g\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:36:15.173811 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:15.173776 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf"] Nov 02 08:36:15.178436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:15.178401 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00cc0e-649cc67649-s6tnf"] Nov 02 08:36:15.585529 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:15.585498 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52606d0f-465b-487e-85ea-d6a7f45256ba" path="/var/lib/kubelet/pods/52606d0f-465b-487e-85ea-d6a7f45256ba/volumes" Nov 02 08:36:15.622397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:15.622365 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:15.858154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:15.858054 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:16.015634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:16.015596 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:18.689130 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:18.689101 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:36:19.697683 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:19.697650 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:36:21.171019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:21.170973 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:22.060380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:22.060342 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:22.499974 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:22.499932 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:22.798205 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:22.798095 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:23.361375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:23.361332 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:26.275574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:26.275532 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:26.503032 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:26.502995 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:26.685702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:26.685664 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:27.049722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:27.049629 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:27.216977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:27.216941 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:27.392135 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:27.392047 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:30.275556 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:30.275515 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:30.489114 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:30.489080 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:31.879492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:31.879449 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:32.793134 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:32.793100 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:33.051208 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:33.051075 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:33.240267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:33.236732 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:37.087157 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:37.087101 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:40.522929 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:36:40.522891 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:36:41.893334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:41.893303 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:36:41.898129 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:41.898109 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:36:42.092416 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:42.092386 6160 scope.go:122] "RemoveContainer" containerID="07df5ada0c8076b67a26a3ecf1398650c2f18e9caa38e12967eb9655e424a303" Nov 02 08:36:42.205784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:42.205748 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:47.387302 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:47.387241 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:50.692512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.692437 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp"] Nov 02 08:36:50.693209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.693193 6160 cpu_manager.go:414] "containerMap: removing container" podUID="52606d0f-465b-487e-85ea-d6a7f45256ba" containerName="payload-processing" Nov 02 08:36:50.693209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.693210 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52606d0f-465b-487e-85ea-d6a7f45256ba" containerName="payload-processing" Nov 02 08:36:50.693319 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.693310 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="52606d0f-465b-487e-85ea-d6a7f45256ba" containerName="payload-processing" Nov 02 08:36:50.698556 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.698538 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.741461 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.741420 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.741627 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.741486 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59jb\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-kube-api-access-r59jb\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.741627 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.741614 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d1d5cd-6b48-41b0-8912-83a69debe723-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.741728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.741654 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.741784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.741739 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.741844 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.741783 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.741937 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.741922 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.741975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.741960 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.742009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.741991 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.798121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.798086 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp"] Nov 02 08:36:50.843839 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.843795 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d1d5cd-6b48-41b0-8912-83a69debe723-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.844034 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.843846 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.844034 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.843920 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.844191 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.844046 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.844267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.844220 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.844327 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.844290 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.844389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.844325 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.844442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.844407 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.844496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.844462 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59jb\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-kube-api-access-r59jb\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.844578 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.844557 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d1d5cd-6b48-41b0-8912-83a69debe723-istiod-ca-cert\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.845500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.845471 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-credential-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.845671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.845479 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-socket\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.845671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.845534 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-certs\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.845671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.845648 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-data\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.846745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.846723 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-envoy\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.847234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.847214 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-podinfo\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.898275 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.898235 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59jb\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-kube-api-access-r59jb\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:50.898409 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:50.898389 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-token\") pod \"e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:51.013397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:51.013306 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:51.438871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:51.438840 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp"] Nov 02 08:36:51.439252 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:36:51.439227 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d1d5cd_6b48_41b0_8912_83a69debe723.slice/crio-3f26bb90048187d3bcdd9d749dded63ac3dde2fcb5b18c09f085b809d56f4e70 WatchSource:0}: Error finding container 3f26bb90048187d3bcdd9d749dded63ac3dde2fcb5b18c09f085b809d56f4e70: Status 404 returned error can't find the container with id 3f26bb90048187d3bcdd9d749dded63ac3dde2fcb5b18c09f085b809d56f4e70 Nov 02 08:36:51.441921 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:51.441882 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:36:51.442035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:51.441966 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:36:51.442035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:51.442018 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:36:51.580401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:51.580340 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:52.055521 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:52.055478 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" event={"ID":"d6d1d5cd-6b48-41b0-8912-83a69debe723","Type":"ContainerStarted","Data":"0c67e814deb57dffd54e0e055461ef35c60b8a0ceba40311f2e4af7b1a41fc74"} Nov 02 08:36:52.055942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:52.055549 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" event={"ID":"d6d1d5cd-6b48-41b0-8912-83a69debe723","Type":"ContainerStarted","Data":"3f26bb90048187d3bcdd9d749dded63ac3dde2fcb5b18c09f085b809d56f4e70"} Nov 02 08:36:52.229200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:52.224609 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" podStartSLOduration=2.22458767 podStartE2EDuration="2.22458767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:52.222558275 +0000 UTC m=+1810.776062533" watchObservedRunningTime="2026-11-02 08:36:52.22458767 +0000 UTC m=+1810.778091871" Nov 02 08:36:52.632462 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:52.632385 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:53.018478 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:53.018445 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:53.022211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:53.018857 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:36:53.022469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:53.022436 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 08:36:54.021108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:54.020632 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:36:54.021108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:54.020709 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 08:36:55.031319 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:55.031268 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:36:55.031813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:55.031357 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 08:36:55.539638 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:55.539598 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn"] Nov 02 08:36:55.831946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:55.831830 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:55.983662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:55.983629 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:56.017628 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:56.017592 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:36:56.017905 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:56.017876 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 08:36:57.014454 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:57.014418 6160 patch_prober.go:29] interesting pod/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:36:57.015073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:57.015038 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 08:36:58.018639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:58.018607 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:58.019046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:58.019007 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:58.019719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:58.019697 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:36:58.215245 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:36:58.215213 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:00.540861 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:00.540803 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" podUID="0ee121a2-369e-435c-ac8a-bee177b9a246" containerName="payload-pre-processing" containerID="cri-o://ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837" gracePeriod=25 Nov 02 08:37:00.846221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:00.846195 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:37:00.921822 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:00.921788 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee121a2-369e-435c-ac8a-bee177b9a246-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee121a2-369e-435c-ac8a-bee177b9a246-plugins-config-volume\") pod \"0ee121a2-369e-435c-ac8a-bee177b9a246\" (UID: \"0ee121a2-369e-435c-ac8a-bee177b9a246\") " Nov 02 08:37:00.921996 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:00.921925 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee121a2-369e-435c-ac8a-bee177b9a246-kube-api-access-fsf6d\" (UniqueName: \"kubernetes.io/projected/0ee121a2-369e-435c-ac8a-bee177b9a246-kube-api-access-fsf6d\") pod \"0ee121a2-369e-435c-ac8a-bee177b9a246\" (UID: \"0ee121a2-369e-435c-ac8a-bee177b9a246\") " Nov 02 08:37:00.922162 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:00.922139 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee121a2-369e-435c-ac8a-bee177b9a246-plugins-config-volume" pod "0ee121a2-369e-435c-ac8a-bee177b9a246" (UID: "0ee121a2-369e-435c-ac8a-bee177b9a246"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:00.922517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:00.922500 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee121a2-369e-435c-ac8a-bee177b9a246-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:00.924234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:00.924210 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee121a2-369e-435c-ac8a-bee177b9a246-kube-api-access-fsf6d" pod "0ee121a2-369e-435c-ac8a-bee177b9a246" (UID: "0ee121a2-369e-435c-ac8a-bee177b9a246"). InnerVolumeSpecName "kube-api-access-fsf6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:01.023655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.023620 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsf6d\" (UniqueName: \"kubernetes.io/projected/0ee121a2-369e-435c-ac8a-bee177b9a246-kube-api-access-fsf6d\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:01.136773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.136683 6160 generic.go:356] "Generic (PLEG): container finished" podID="0ee121a2-369e-435c-ac8a-bee177b9a246" containerID="ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837" exitCode=0 Nov 02 08:37:01.136773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.136746 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" Nov 02 08:37:01.137065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.136767 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" event={"ID":"0ee121a2-369e-435c-ac8a-bee177b9a246","Type":"ContainerDied","Data":"ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837"} Nov 02 08:37:01.137065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.136850 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn" event={"ID":"0ee121a2-369e-435c-ac8a-bee177b9a246","Type":"ContainerDied","Data":"0f4c74b1f0fa1b8eb51484bc82285d44684219337a714d82848edfadedbe6367"} Nov 02 08:37:01.137065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.136873 6160 scope.go:122] "RemoveContainer" containerID="ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837" Nov 02 08:37:01.147921 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.147904 6160 scope.go:122] "RemoveContainer" containerID="ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837" Nov 02 08:37:01.148285 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:37:01.148264 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837\": container with ID starting with ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837 not found: ID does not exist" containerID="ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837" Nov 02 08:37:01.148344 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.148294 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837"} err="failed to get container status \"ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837\": rpc error: code = NotFound desc = could not find container \"ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837\": container with ID starting with ad7c6d461242bb3604d0cc2481270b61c338a8ae906af89f138dce4d4efb5837 not found: ID does not exist" Nov 02 08:37:01.291143 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.291112 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn"] Nov 02 08:37:01.294921 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.294897 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c7f7d7-67b6db8f55-s5znn"] Nov 02 08:37:01.586551 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:01.586517 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee121a2-369e-435c-ac8a-bee177b9a246" path="/var/lib/kubelet/pods/0ee121a2-369e-435c-ac8a-bee177b9a246/volumes" Nov 02 08:37:03.051997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:03.051955 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:07.966437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:07.966403 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:10.522593 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:37:10.522555 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:37:11.959423 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:11.959390 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf"] Nov 02 08:37:11.959846 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:11.959600 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" podUID="bec2d528-ffaa-48a0-91a0-15b7a85c7e44" containerName="maas-api" containerID="cri-o://d8679e1b3d90c2a32dce855902b41b2e4fc39460ddcacd971b97d7c3bf41de62" gracePeriod=30 Nov 02 08:37:12.078666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.078631 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:12.227348 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.227314 6160 generic.go:356] "Generic (PLEG): container finished" podID="bec2d528-ffaa-48a0-91a0-15b7a85c7e44" containerID="d8679e1b3d90c2a32dce855902b41b2e4fc39460ddcacd971b97d7c3bf41de62" exitCode=0 Nov 02 08:37:12.227491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.227377 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" event={"ID":"bec2d528-ffaa-48a0-91a0-15b7a85c7e44","Type":"ContainerDied","Data":"d8679e1b3d90c2a32dce855902b41b2e4fc39460ddcacd971b97d7c3bf41de62"} Nov 02 08:37:12.257958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.257930 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:37:12.403066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.391761 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-metrics-certs\") pod \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " Nov 02 08:37:12.403066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.391873 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-kube-api-access-8dq7b\" (UniqueName: \"kubernetes.io/projected/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-kube-api-access-8dq7b\") pod \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " Nov 02 08:37:12.403066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.391960 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls\") pod \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " Nov 02 08:37:12.403066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.394640 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-openshift-service-ca\") pod \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\" (UID: \"bec2d528-ffaa-48a0-91a0-15b7a85c7e44\") " Nov 02 08:37:12.403066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.396880 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-metrics-certs" pod "bec2d528-ffaa-48a0-91a0-15b7a85c7e44" (UID: "bec2d528-ffaa-48a0-91a0-15b7a85c7e44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:12.403066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.397101 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-openshift-service-ca" pod "bec2d528-ffaa-48a0-91a0-15b7a85c7e44" (UID: "bec2d528-ffaa-48a0-91a0-15b7a85c7e44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:12.404234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.404080 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-kube-api-access-8dq7b" pod "bec2d528-ffaa-48a0-91a0-15b7a85c7e44" (UID: "bec2d528-ffaa-48a0-91a0-15b7a85c7e44"). InnerVolumeSpecName "kube-api-access-8dq7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:12.439366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.439326 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls" pod "bec2d528-ffaa-48a0-91a0-15b7a85c7e44" (UID: "bec2d528-ffaa-48a0-91a0-15b7a85c7e44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:12.489790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.489709 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698"] Nov 02 08:37:12.489949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.489899 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" podUID="4add2425-f901-4880-bb0e-a74a7d44d33e" containerName="payload-pre-processing" containerID="cri-o://2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb" gracePeriod=30 Nov 02 08:37:12.497406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.497381 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dq7b\" (UniqueName: \"kubernetes.io/projected/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-kube-api-access-8dq7b\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:12.497519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.497412 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:12.497519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.497426 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:12.497519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.497447 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec2d528-ffaa-48a0-91a0-15b7a85c7e44-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:12.623899 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.623853 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:12.742221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.742143 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:37:12.828002 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.827967 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:12.904539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.904510 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4add2425-f901-4880-bb0e-a74a7d44d33e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4add2425-f901-4880-bb0e-a74a7d44d33e-plugins-config-volume\") pod \"4add2425-f901-4880-bb0e-a74a7d44d33e\" (UID: \"4add2425-f901-4880-bb0e-a74a7d44d33e\") " Nov 02 08:37:12.904728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.904547 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4add2425-f901-4880-bb0e-a74a7d44d33e-kube-api-access-z4nvv\" (UniqueName: \"kubernetes.io/projected/4add2425-f901-4880-bb0e-a74a7d44d33e-kube-api-access-z4nvv\") pod \"4add2425-f901-4880-bb0e-a74a7d44d33e\" (UID: \"4add2425-f901-4880-bb0e-a74a7d44d33e\") " Nov 02 08:37:12.904977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.904946 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4add2425-f901-4880-bb0e-a74a7d44d33e-plugins-config-volume" pod "4add2425-f901-4880-bb0e-a74a7d44d33e" (UID: "4add2425-f901-4880-bb0e-a74a7d44d33e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:12.905225 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.905168 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4add2425-f901-4880-bb0e-a74a7d44d33e-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:12.906918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:12.906893 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4add2425-f901-4880-bb0e-a74a7d44d33e-kube-api-access-z4nvv" pod "4add2425-f901-4880-bb0e-a74a7d44d33e" (UID: "4add2425-f901-4880-bb0e-a74a7d44d33e"). InnerVolumeSpecName "kube-api-access-z4nvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:13.023933 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.023901 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4nvv\" (UniqueName: \"kubernetes.io/projected/4add2425-f901-4880-bb0e-a74a7d44d33e-kube-api-access-z4nvv\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:13.235742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.235710 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" Nov 02 08:37:13.235936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.235709 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf" event={"ID":"bec2d528-ffaa-48a0-91a0-15b7a85c7e44","Type":"ContainerDied","Data":"9e8685dff4f9b41d8a7cbcad55ef16d926755f33bebee9853335cc49629136aa"} Nov 02 08:37:13.235936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.235874 6160 scope.go:122] "RemoveContainer" containerID="d8679e1b3d90c2a32dce855902b41b2e4fc39460ddcacd971b97d7c3bf41de62" Nov 02 08:37:13.237304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.237154 6160 generic.go:356] "Generic (PLEG): container finished" podID="4add2425-f901-4880-bb0e-a74a7d44d33e" containerID="2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb" exitCode=0 Nov 02 08:37:13.237304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.237243 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" Nov 02 08:37:13.237304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.237241 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" event={"ID":"4add2425-f901-4880-bb0e-a74a7d44d33e","Type":"ContainerDied","Data":"2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb"} Nov 02 08:37:13.237476 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.237351 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698" event={"ID":"4add2425-f901-4880-bb0e-a74a7d44d33e","Type":"ContainerDied","Data":"4eaecf068c45d84db89cce4232fb36b398aabd56d47b0e363a95ddded60156db"} Nov 02 08:37:13.248915 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.248894 6160 scope.go:122] "RemoveContainer" containerID="2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb" Nov 02 08:37:13.259382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.259361 6160 scope.go:122] "RemoveContainer" containerID="2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb" Nov 02 08:37:13.259670 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:37:13.259653 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb\": container with ID starting with 2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb not found: ID does not exist" containerID="2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb" Nov 02 08:37:13.259719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.259680 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb"} err="failed to get container status \"2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb\": rpc error: code = NotFound desc = could not find container \"2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb\": container with ID starting with 2448295ec5f8db9778c11c73d314a229c6f9dca3638a3b9f5d901782c31f31cb not found: ID does not exist" Nov 02 08:37:13.392709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.392617 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf"] Nov 02 08:37:13.396362 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.396339 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bca68b-5967898f7d-wfnkf"] Nov 02 08:37:13.497264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.497220 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698"] Nov 02 08:37:13.501090 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.501060 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bca68b-dbd9bf47f-tk698"] Nov 02 08:37:13.587732 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.587699 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4add2425-f901-4880-bb0e-a74a7d44d33e" path="/var/lib/kubelet/pods/4add2425-f901-4880-bb0e-a74a7d44d33e/volumes" Nov 02 08:37:13.588211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.588170 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec2d528-ffaa-48a0-91a0-15b7a85c7e44" path="/var/lib/kubelet/pods/bec2d528-ffaa-48a0-91a0-15b7a85c7e44/volumes" Nov 02 08:37:13.805412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:13.805359 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:14.015244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:14.015206 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:14.165100 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:14.165062 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:14.665471 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:14.665435 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:19.263112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:19.263072 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:20.463123 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:20.463082 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:20.630741 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:20.630708 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:20.817231 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:20.817131 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:22.849311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:22.849270 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:23.818540 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:23.818499 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:25.533880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:25.533848 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:25.856013 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:25.855914 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:26.053430 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:26.053384 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:26.473403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:26.473350 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:32.078041 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:32.078006 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:32.234722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:32.234690 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:32.370630 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:32.370534 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d"] Nov 02 08:37:32.370828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:32.370798 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" containerID="cri-o://c00e39643974a83d43bfaafff0d25eb620af65f32c612d175e173ec4ddc09b23" gracePeriod=30 Nov 02 08:37:32.409273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:32.409236 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:32.570741 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:32.570706 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:33.185252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:33.185216 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:33.392753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:33.392720 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:33.767547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:33.767508 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc"] Nov 02 08:37:33.767735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:33.767720 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" podUID="bcb05d8b-3106-42b3-b257-eb1fb643fd0d" containerName="maas-api" containerID="cri-o://efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6" gracePeriod=30 Nov 02 08:37:33.853821 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:33.853790 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx"] Nov 02 08:37:33.918869 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:33.918580 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:33.951161 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:33.951116 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t"] Nov 02 08:37:34.090955 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.090923 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:37:34.096815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.096781 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:34.158985 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.158942 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-metrics-certs\") pod \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " Nov 02 08:37:34.159188 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.159032 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-openshift-service-ca\") pod \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " Nov 02 08:37:34.159188 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.159080 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-kube-api-access-b4hrm\" (UniqueName: \"kubernetes.io/projected/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-kube-api-access-b4hrm\") pod \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " Nov 02 08:37:34.159317 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.159231 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls\") pod \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\" (UID: \"bcb05d8b-3106-42b3-b257-eb1fb643fd0d\") " Nov 02 08:37:34.159466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.159438 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-openshift-service-ca" pod "bcb05d8b-3106-42b3-b257-eb1fb643fd0d" (UID: "bcb05d8b-3106-42b3-b257-eb1fb643fd0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:34.160670 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.160625 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:34.162222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.162161 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls" pod "bcb05d8b-3106-42b3-b257-eb1fb643fd0d" (UID: "bcb05d8b-3106-42b3-b257-eb1fb643fd0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:34.163145 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.163108 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-kube-api-access-b4hrm" pod "bcb05d8b-3106-42b3-b257-eb1fb643fd0d" (UID: "bcb05d8b-3106-42b3-b257-eb1fb643fd0d"). InnerVolumeSpecName "kube-api-access-b4hrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:34.166197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.164990 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-metrics-certs" pod "bcb05d8b-3106-42b3-b257-eb1fb643fd0d" (UID: "bcb05d8b-3106-42b3-b257-eb1fb643fd0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:34.263198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.263011 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:34.263198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.263046 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:34.263198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.263061 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4hrm\" (UniqueName: \"kubernetes.io/projected/bcb05d8b-3106-42b3-b257-eb1fb643fd0d-kube-api-access-b4hrm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:34.331358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.330839 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:34.494828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.494794 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:34.497794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.497767 6160 generic.go:356] "Generic (PLEG): container finished" podID="bcb05d8b-3106-42b3-b257-eb1fb643fd0d" containerID="efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6" exitCode=0 Nov 02 08:37:34.497915 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.497813 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" Nov 02 08:37:34.497915 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.497843 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" event={"ID":"bcb05d8b-3106-42b3-b257-eb1fb643fd0d","Type":"ContainerDied","Data":"efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6"} Nov 02 08:37:34.497915 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.497900 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc" event={"ID":"bcb05d8b-3106-42b3-b257-eb1fb643fd0d","Type":"ContainerDied","Data":"e88740171d50672bf6ab0567355443a1024ce921339d27e0387c5a86ae253e7b"} Nov 02 08:37:34.498051 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.497929 6160 scope.go:122] "RemoveContainer" containerID="efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6" Nov 02 08:37:34.513553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.513525 6160 scope.go:122] "RemoveContainer" containerID="efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6" Nov 02 08:37:34.514005 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:37:34.513977 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6\": container with ID starting with efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6 not found: ID does not exist" containerID="efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6" Nov 02 08:37:34.514091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.514021 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6"} err="failed to get container status \"efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6\": rpc error: code = NotFound desc = could not find container \"efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6\": container with ID starting with efd73489a06e2c2a3b131223a2cff9f359a143e84cea36ae41c1333523ba82d6 not found: ID does not exist" Nov 02 08:37:34.653876 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.653775 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:34.656650 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.656596 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc"] Nov 02 08:37:34.660307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:34.660287 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c17125-6b95cb7878-ncmgc"] Nov 02 08:37:35.277068 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:35.277033 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:35.586049 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:35.585973 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb05d8b-3106-42b3-b257-eb1fb643fd0d" path="/var/lib/kubelet/pods/bcb05d8b-3106-42b3-b257-eb1fb643fd0d/volumes" Nov 02 08:37:35.841053 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:35.840957 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:37.525799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.525766 6160 generic.go:356] "Generic (PLEG): container finished" podID="3e44e452-05c5-4715-801d-ffa370eadee1" containerID="c00e39643974a83d43bfaafff0d25eb620af65f32c612d175e173ec4ddc09b23" exitCode=0 Nov 02 08:37:37.526272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.525842 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" event={"ID":"3e44e452-05c5-4715-801d-ffa370eadee1","Type":"ContainerDied","Data":"c00e39643974a83d43bfaafff0d25eb620af65f32c612d175e173ec4ddc09b23"} Nov 02 08:37:37.526272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.525920 6160 scope.go:122] "RemoveContainer" containerID="516cb709cdb3f8e98b3fa1c6e417f6e26d68f56e86810e28dc5f16ec137f7332" Nov 02 08:37:37.661824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.661796 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:37:37.712097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712064 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-kube-api-access-n4vnn\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-kube-api-access-n4vnn\") pod \"3e44e452-05c5-4715-801d-ffa370eadee1\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " Nov 02 08:37:37.712313 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712112 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-envoy\") pod \"3e44e452-05c5-4715-801d-ffa370eadee1\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " Nov 02 08:37:37.712313 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712141 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-socket\") pod \"3e44e452-05c5-4715-801d-ffa370eadee1\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " Nov 02 08:37:37.712313 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712208 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-certs\") pod \"3e44e452-05c5-4715-801d-ffa370eadee1\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " Nov 02 08:37:37.712313 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712240 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e44e452-05c5-4715-801d-ffa370eadee1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e452-05c5-4715-801d-ffa370eadee1-istiod-ca-cert\") pod \"3e44e452-05c5-4715-801d-ffa370eadee1\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " Nov 02 08:37:37.712532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712329 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-istio-token\") pod \"3e44e452-05c5-4715-801d-ffa370eadee1\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " Nov 02 08:37:37.712532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712395 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-data\") pod \"3e44e452-05c5-4715-801d-ffa370eadee1\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " Nov 02 08:37:37.712532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712464 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-socket" pod "3e44e452-05c5-4715-801d-ffa370eadee1" (UID: "3e44e452-05c5-4715-801d-ffa370eadee1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:37.712723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712573 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-certs" pod "3e44e452-05c5-4715-801d-ffa370eadee1" (UID: "3e44e452-05c5-4715-801d-ffa370eadee1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:37.712723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712612 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-credential-socket\") pod \"3e44e452-05c5-4715-801d-ffa370eadee1\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " Nov 02 08:37:37.712723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712629 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e44e452-05c5-4715-801d-ffa370eadee1-istiod-ca-cert" pod "3e44e452-05c5-4715-801d-ffa370eadee1" (UID: "3e44e452-05c5-4715-801d-ffa370eadee1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:37.712723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712694 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e44e452-05c5-4715-801d-ffa370eadee1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e452-05c5-4715-801d-ffa370eadee1-istio-podinfo\") pod \"3e44e452-05c5-4715-801d-ffa370eadee1\" (UID: \"3e44e452-05c5-4715-801d-ffa370eadee1\") " Nov 02 08:37:37.712918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.712746 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-data" pod "3e44e452-05c5-4715-801d-ffa370eadee1" (UID: "3e44e452-05c5-4715-801d-ffa370eadee1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:37.713100 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.713074 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-credential-socket" pod "3e44e452-05c5-4715-801d-ffa370eadee1" (UID: "3e44e452-05c5-4715-801d-ffa370eadee1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:37.713350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.713328 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:37.713450 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.713357 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:37.713450 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.713375 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:37.713450 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.713388 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e452-05c5-4715-801d-ffa370eadee1-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:37.713450 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.713402 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:37.714623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.714595 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-envoy" pod "3e44e452-05c5-4715-801d-ffa370eadee1" (UID: "3e44e452-05c5-4715-801d-ffa370eadee1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:37.714729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.714666 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-istio-token" pod "3e44e452-05c5-4715-801d-ffa370eadee1" (UID: "3e44e452-05c5-4715-801d-ffa370eadee1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:37.714729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.714703 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-kube-api-access-n4vnn" pod "3e44e452-05c5-4715-801d-ffa370eadee1" (UID: "3e44e452-05c5-4715-801d-ffa370eadee1"). InnerVolumeSpecName "kube-api-access-n4vnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:37.715086 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.715071 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e44e452-05c5-4715-801d-ffa370eadee1-istio-podinfo" pod "3e44e452-05c5-4715-801d-ffa370eadee1" (UID: "3e44e452-05c5-4715-801d-ffa370eadee1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:37:37.814652 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.814611 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:37.814652 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.814642 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e452-05c5-4715-801d-ffa370eadee1-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:37.814652 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.814656 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4vnn\" (UniqueName: \"kubernetes.io/projected/3e44e452-05c5-4715-801d-ffa370eadee1-kube-api-access-n4vnn\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:37.814883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:37.814671 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e452-05c5-4715-801d-ffa370eadee1-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:38.419946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:38.419909 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:38.534950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:38.534919 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" Nov 02 08:37:38.535435 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:38.534948 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d" event={"ID":"3e44e452-05c5-4715-801d-ffa370eadee1","Type":"ContainerDied","Data":"527d86217def818d0d77ac3239dfb32fe78cc2e88c41b3bf0af8fc7b50b2e5d7"} Nov 02 08:37:38.535435 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:38.535009 6160 scope.go:122] "RemoveContainer" containerID="c00e39643974a83d43bfaafff0d25eb620af65f32c612d175e173ec4ddc09b23" Nov 02 08:37:38.739145 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:38.739041 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d"] Nov 02 08:37:38.761791 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:38.761752 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:38.786841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:38.786802 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a25202-openshift-default-745bcccd99-d7c9d"] Nov 02 08:37:38.854542 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:38.854503 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" podUID="91956a2d-4b70-4d16-85ac-757af9bdf895" containerName="payload-processing" containerID="cri-o://b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39" gracePeriod=25 Nov 02 08:37:38.951751 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:38.951688 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" podUID="31a63621-ee13-463b-a2c0-50862c92ca21" containerName="payload-pre-processing" containerID="cri-o://86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7" gracePeriod=25 Nov 02 08:37:39.153835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.153801 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:37:39.221220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.221189 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:37:39.233489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.233449 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91956a2d-4b70-4d16-85ac-757af9bdf895-kube-api-access-mhsg7\" (UniqueName: \"kubernetes.io/projected/91956a2d-4b70-4d16-85ac-757af9bdf895-kube-api-access-mhsg7\") pod \"91956a2d-4b70-4d16-85ac-757af9bdf895\" (UID: \"91956a2d-4b70-4d16-85ac-757af9bdf895\") " Nov 02 08:37:39.233653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.233505 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91956a2d-4b70-4d16-85ac-757af9bdf895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91956a2d-4b70-4d16-85ac-757af9bdf895-plugins-config-volume\") pod \"91956a2d-4b70-4d16-85ac-757af9bdf895\" (UID: \"91956a2d-4b70-4d16-85ac-757af9bdf895\") " Nov 02 08:37:39.233884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.233861 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91956a2d-4b70-4d16-85ac-757af9bdf895-plugins-config-volume" pod "91956a2d-4b70-4d16-85ac-757af9bdf895" (UID: "91956a2d-4b70-4d16-85ac-757af9bdf895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:39.234217 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.234198 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91956a2d-4b70-4d16-85ac-757af9bdf895-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:39.235735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.235712 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91956a2d-4b70-4d16-85ac-757af9bdf895-kube-api-access-mhsg7" pod "91956a2d-4b70-4d16-85ac-757af9bdf895" (UID: "91956a2d-4b70-4d16-85ac-757af9bdf895"). InnerVolumeSpecName "kube-api-access-mhsg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:39.335224 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.335111 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a63621-ee13-463b-a2c0-50862c92ca21-kube-api-access-qbpwp\" (UniqueName: \"kubernetes.io/projected/31a63621-ee13-463b-a2c0-50862c92ca21-kube-api-access-qbpwp\") pod \"31a63621-ee13-463b-a2c0-50862c92ca21\" (UID: \"31a63621-ee13-463b-a2c0-50862c92ca21\") " Nov 02 08:37:39.335406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.335247 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a63621-ee13-463b-a2c0-50862c92ca21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a63621-ee13-463b-a2c0-50862c92ca21-plugins-config-volume\") pod \"31a63621-ee13-463b-a2c0-50862c92ca21\" (UID: \"31a63621-ee13-463b-a2c0-50862c92ca21\") " Nov 02 08:37:39.335646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.335621 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a63621-ee13-463b-a2c0-50862c92ca21-plugins-config-volume" pod "31a63621-ee13-463b-a2c0-50862c92ca21" (UID: "31a63621-ee13-463b-a2c0-50862c92ca21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:39.335754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.335629 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhsg7\" (UniqueName: \"kubernetes.io/projected/91956a2d-4b70-4d16-85ac-757af9bdf895-kube-api-access-mhsg7\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:39.337399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.337376 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a63621-ee13-463b-a2c0-50862c92ca21-kube-api-access-qbpwp" pod "31a63621-ee13-463b-a2c0-50862c92ca21" (UID: "31a63621-ee13-463b-a2c0-50862c92ca21"). InnerVolumeSpecName "kube-api-access-qbpwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:39.355813 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.355782 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:39.436930 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.436820 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbpwp\" (UniqueName: \"kubernetes.io/projected/31a63621-ee13-463b-a2c0-50862c92ca21-kube-api-access-qbpwp\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:39.436930 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.436858 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a63621-ee13-463b-a2c0-50862c92ca21-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:39.544125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.544092 6160 generic.go:356] "Generic (PLEG): container finished" podID="31a63621-ee13-463b-a2c0-50862c92ca21" containerID="86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7" exitCode=0 Nov 02 08:37:39.544657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.544149 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" Nov 02 08:37:39.544657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.544199 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" event={"ID":"31a63621-ee13-463b-a2c0-50862c92ca21","Type":"ContainerDied","Data":"86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7"} Nov 02 08:37:39.544657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.544296 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t" event={"ID":"31a63621-ee13-463b-a2c0-50862c92ca21","Type":"ContainerDied","Data":"6393eebaa673545009e7d1e79c9171ab95adc7c4edf9ead17768dab5244d9bb8"} Nov 02 08:37:39.544657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.544322 6160 scope.go:122] "RemoveContainer" containerID="86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7" Nov 02 08:37:39.546860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.546839 6160 generic.go:356] "Generic (PLEG): container finished" podID="91956a2d-4b70-4d16-85ac-757af9bdf895" containerID="b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39" exitCode=0 Nov 02 08:37:39.546994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.546884 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" Nov 02 08:37:39.546994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.546937 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" event={"ID":"91956a2d-4b70-4d16-85ac-757af9bdf895","Type":"ContainerDied","Data":"b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39"} Nov 02 08:37:39.546994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.546979 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx" event={"ID":"91956a2d-4b70-4d16-85ac-757af9bdf895","Type":"ContainerDied","Data":"098036b38f9d7d7092341ee92a2086e954069282582893bbdf6c2354e2d65bba"} Nov 02 08:37:39.557292 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.557275 6160 scope.go:122] "RemoveContainer" containerID="86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7" Nov 02 08:37:39.557566 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:37:39.557545 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7\": container with ID starting with 86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7 not found: ID does not exist" containerID="86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7" Nov 02 08:37:39.557626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.557574 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7"} err="failed to get container status \"86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7\": rpc error: code = NotFound desc = could not find container \"86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7\": container with ID starting with 86fa69bf80924f92ae4b3bf22d16d0b3e3e7417989f40cf82d7167facd5fb6b7 not found: ID does not exist" Nov 02 08:37:39.557626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.557591 6160 scope.go:122] "RemoveContainer" containerID="b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39" Nov 02 08:37:39.571667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.571640 6160 scope.go:122] "RemoveContainer" containerID="b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39" Nov 02 08:37:39.572255 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:37:39.572231 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39\": container with ID starting with b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39 not found: ID does not exist" containerID="b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39" Nov 02 08:37:39.572349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.572263 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39"} err="failed to get container status \"b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39\": rpc error: code = NotFound desc = could not find container \"b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39\": container with ID starting with b1c9731e94b05493aa61a2f976aa8f654cdb43e5f0379d6e977539690977bb39 not found: ID does not exist" Nov 02 08:37:39.577911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.577884 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:39.586096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.586042 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" path="/var/lib/kubelet/pods/3e44e452-05c5-4715-801d-ffa370eadee1/volumes" Nov 02 08:37:39.700089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.700053 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t"] Nov 02 08:37:39.703665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.703636 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c17125-79ccd5bfbc-k745t"] Nov 02 08:37:39.804492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.804453 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx"] Nov 02 08:37:39.810672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:39.810642 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c17125-c9cfb7b45-fzdvx"] Nov 02 08:37:40.523037 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:37:40.523000 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:37:40.922463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:40.922411 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:41.090268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:41.090229 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:41.249065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:41.248972 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:41.516574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:41.516487 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:41.698344 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:41.698307 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a63621-ee13-463b-a2c0-50862c92ca21" path="/var/lib/kubelet/pods/31a63621-ee13-463b-a2c0-50862c92ca21/volumes" Nov 02 08:37:41.698962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:41.698941 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91956a2d-4b70-4d16-85ac-757af9bdf895" path="/var/lib/kubelet/pods/91956a2d-4b70-4d16-85ac-757af9bdf895/volumes" Nov 02 08:37:44.261226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:44.261169 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:44.467968 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:44.467929 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:46.522553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:46.522514 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:47.792196 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:47.792151 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:49.417692 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:49.417651 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:51.365370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:51.365331 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:53.854880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:53.854847 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769"] Nov 02 08:37:53.973629 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:53.973598 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk"] Nov 02 08:37:54.260044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:54.260008 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:54.462978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:54.462937 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:55.383050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.382999 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258"] Nov 02 08:37:55.383691 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383674 6160 cpu_manager.go:414] "containerMap: removing container" podUID="bec2d528-ffaa-48a0-91a0-15b7a85c7e44" containerName="maas-api" Nov 02 08:37:55.383739 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383693 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec2d528-ffaa-48a0-91a0-15b7a85c7e44" containerName="maas-api" Nov 02 08:37:55.383739 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383723 6160 cpu_manager.go:414] "containerMap: removing container" podUID="4add2425-f901-4880-bb0e-a74a7d44d33e" containerName="payload-pre-processing" Nov 02 08:37:55.383739 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383728 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4add2425-f901-4880-bb0e-a74a7d44d33e" containerName="payload-pre-processing" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383744 6160 cpu_manager.go:414] "containerMap: removing container" podUID="31a63621-ee13-463b-a2c0-50862c92ca21" containerName="payload-pre-processing" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383749 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a63621-ee13-463b-a2c0-50862c92ca21" containerName="payload-pre-processing" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383762 6160 cpu_manager.go:414] "containerMap: removing container" podUID="bcb05d8b-3106-42b3-b257-eb1fb643fd0d" containerName="maas-api" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383767 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb05d8b-3106-42b3-b257-eb1fb643fd0d" containerName="maas-api" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383776 6160 cpu_manager.go:414] "containerMap: removing container" podUID="0ee121a2-369e-435c-ac8a-bee177b9a246" containerName="payload-pre-processing" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383780 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee121a2-369e-435c-ac8a-bee177b9a246" containerName="payload-pre-processing" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383790 6160 cpu_manager.go:414] "containerMap: removing container" podUID="91956a2d-4b70-4d16-85ac-757af9bdf895" containerName="payload-processing" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383794 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91956a2d-4b70-4d16-85ac-757af9bdf895" containerName="payload-processing" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383802 6160 cpu_manager.go:414] "containerMap: removing container" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383807 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383814 6160 cpu_manager.go:414] "containerMap: removing container" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" Nov 02 08:37:55.383833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383818 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" Nov 02 08:37:55.384228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383902 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec2d528-ffaa-48a0-91a0-15b7a85c7e44" containerName="maas-api" Nov 02 08:37:55.384228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383913 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="4add2425-f901-4880-bb0e-a74a7d44d33e" containerName="payload-pre-processing" Nov 02 08:37:55.384228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383921 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a63621-ee13-463b-a2c0-50862c92ca21" containerName="payload-pre-processing" Nov 02 08:37:55.384228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383928 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb05d8b-3106-42b3-b257-eb1fb643fd0d" containerName="maas-api" Nov 02 08:37:55.384228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383933 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee121a2-369e-435c-ac8a-bee177b9a246" containerName="payload-pre-processing" Nov 02 08:37:55.384228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383939 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="91956a2d-4b70-4d16-85ac-757af9bdf895" containerName="payload-processing" Nov 02 08:37:55.384228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383945 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" Nov 02 08:37:55.384228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.383951 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e44e452-05c5-4715-801d-ffa370eadee1" containerName="istio-proxy" Nov 02 08:37:55.387547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.387530 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.437230 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.437199 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cba6f9af-openshift-default-dockercfg-lc888\"" Nov 02 08:37:55.482156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.482117 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvdf\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-kube-api-access-tdvdf\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.482464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.482438 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.482630 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.482613 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.482828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.482811 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.482952 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.482937 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.483059 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.483044 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.483223 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.483207 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.483452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.483434 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.483579 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.483564 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.496632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.496592 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258"] Nov 02 08:37:55.498451 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.498420 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:55.585294 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.585257 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvdf\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-kube-api-access-tdvdf\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.585584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.585561 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.586085 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.586060 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.586378 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.586315 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.586568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.586548 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.586717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.586697 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.586898 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.586878 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.587131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.587114 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.587377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.587348 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.587531 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.587505 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.587657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.586370 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.587657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.586123 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.588229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.588208 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.588923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.588895 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.592417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.592355 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.612197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.610160 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.651993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.651871 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvdf\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-kube-api-access-tdvdf\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.651993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.651910 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-pz258\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:55.675335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.675279 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:55.701805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:55.701746 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:56.006572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:56.006538 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:56.181490 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:37:56.181459 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdce3e7_463e_4150_8e0f_ab3d35dfbfdb.slice/crio-7b3d1394fefa14149aa27dd931d25444612a876d32d28adcd0eab1a8c8a11224 WatchSource:0}: Error finding container 7b3d1394fefa14149aa27dd931d25444612a876d32d28adcd0eab1a8c8a11224: Status 404 returned error can't find the container with id 7b3d1394fefa14149aa27dd931d25444612a876d32d28adcd0eab1a8c8a11224 Nov 02 08:37:56.184256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:56.184220 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:37:56.184340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:56.184294 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:37:56.184340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:56.184303 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258"] Nov 02 08:37:56.184420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:56.184344 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:37:56.804613 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:56.804573 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" event={"ID":"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb","Type":"ContainerStarted","Data":"7a1082c0932fa4a2f2dd1ebd44cb44355179caf5d4f6bead0ffc10136059934c"} Nov 02 08:37:56.805071 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:56.804642 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" event={"ID":"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb","Type":"ContainerStarted","Data":"7b3d1394fefa14149aa27dd931d25444612a876d32d28adcd0eab1a8c8a11224"} Nov 02 08:37:56.920965 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:56.920910 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" podStartSLOduration=1.9208944959999998 podStartE2EDuration="1.920894496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:37:56.918235655 +0000 UTC m=+1875.471739870" watchObservedRunningTime="2026-11-02 08:37:56.920894496 +0000 UTC m=+1875.474398763" Nov 02 08:37:57.702554 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:57.702519 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:37:57.703963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:57.703924 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-pz258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:37:57.704096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:57.703993 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 08:37:58.031034 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:58.030813 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:58.703022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:58.702984 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-pz258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:37:58.703232 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:58.703055 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 08:37:58.855402 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:58.855349 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" podUID="570f3f22-3c1d-4206-b0dd-474023f4e48a" containerName="payload-processing" containerID="cri-o://7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88" gracePeriod=25 Nov 02 08:37:58.974137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:58.974106 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" podUID="c1f3da8a-b765-44b0-b28d-c20f48cbf732" containerName="payload-pre-processing" containerID="cri-o://70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb" gracePeriod=25 Nov 02 08:37:59.184300 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.184274 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:37:59.257688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.256818 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570f3f22-3c1d-4206-b0dd-474023f4e48a-kube-api-access-t5nj5\" (UniqueName: \"kubernetes.io/projected/570f3f22-3c1d-4206-b0dd-474023f4e48a-kube-api-access-t5nj5\") pod \"570f3f22-3c1d-4206-b0dd-474023f4e48a\" (UID: \"570f3f22-3c1d-4206-b0dd-474023f4e48a\") " Nov 02 08:37:59.257688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.256998 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570f3f22-3c1d-4206-b0dd-474023f4e48a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f3f22-3c1d-4206-b0dd-474023f4e48a-plugins-config-volume\") pod \"570f3f22-3c1d-4206-b0dd-474023f4e48a\" (UID: \"570f3f22-3c1d-4206-b0dd-474023f4e48a\") " Nov 02 08:37:59.257688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.257582 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570f3f22-3c1d-4206-b0dd-474023f4e48a-plugins-config-volume" pod "570f3f22-3c1d-4206-b0dd-474023f4e48a" (UID: "570f3f22-3c1d-4206-b0dd-474023f4e48a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:59.257967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.257750 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f3f22-3c1d-4206-b0dd-474023f4e48a-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:59.261649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.261611 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570f3f22-3c1d-4206-b0dd-474023f4e48a-kube-api-access-t5nj5" pod "570f3f22-3c1d-4206-b0dd-474023f4e48a" (UID: "570f3f22-3c1d-4206-b0dd-474023f4e48a"). InnerVolumeSpecName "kube-api-access-t5nj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:59.301926 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.301841 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:37:59.359324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.359292 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f3da8a-b765-44b0-b28d-c20f48cbf732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f3da8a-b765-44b0-b28d-c20f48cbf732-plugins-config-volume\") pod \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\" (UID: \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\") " Nov 02 08:37:59.359465 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.359346 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f3da8a-b765-44b0-b28d-c20f48cbf732-kube-api-access-2txjs\" (UniqueName: \"kubernetes.io/projected/c1f3da8a-b765-44b0-b28d-c20f48cbf732-kube-api-access-2txjs\") pod \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\" (UID: \"c1f3da8a-b765-44b0-b28d-c20f48cbf732\") " Nov 02 08:37:59.359695 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.359672 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f3da8a-b765-44b0-b28d-c20f48cbf732-plugins-config-volume" pod "c1f3da8a-b765-44b0-b28d-c20f48cbf732" (UID: "c1f3da8a-b765-44b0-b28d-c20f48cbf732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:59.359929 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.359909 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5nj5\" (UniqueName: \"kubernetes.io/projected/570f3f22-3c1d-4206-b0dd-474023f4e48a-kube-api-access-t5nj5\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:59.359966 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.359937 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f3da8a-b765-44b0-b28d-c20f48cbf732-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:59.361725 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.361696 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f3da8a-b765-44b0-b28d-c20f48cbf732-kube-api-access-2txjs" pod "c1f3da8a-b765-44b0-b28d-c20f48cbf732" (UID: "c1f3da8a-b765-44b0-b28d-c20f48cbf732"). InnerVolumeSpecName "kube-api-access-2txjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:59.461236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.461204 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txjs\" (UniqueName: \"kubernetes.io/projected/c1f3da8a-b765-44b0-b28d-c20f48cbf732-kube-api-access-2txjs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:37:59.702131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.702101 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-pz258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:37:59.702323 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.702163 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 08:37:59.851290 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.851200 6160 generic.go:356] "Generic (PLEG): container finished" podID="570f3f22-3c1d-4206-b0dd-474023f4e48a" containerID="7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88" exitCode=0 Nov 02 08:37:59.851290 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.851271 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" event={"ID":"570f3f22-3c1d-4206-b0dd-474023f4e48a","Type":"ContainerDied","Data":"7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88"} Nov 02 08:37:59.851524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.851317 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" Nov 02 08:37:59.851524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.851333 6160 scope.go:122] "RemoveContainer" containerID="7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88" Nov 02 08:37:59.851524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.851323 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769" event={"ID":"570f3f22-3c1d-4206-b0dd-474023f4e48a","Type":"ContainerDied","Data":"21b0a68138933576d64899fb06d733caff1e322d6d15b852eb52e77e59f00343"} Nov 02 08:37:59.852881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.852854 6160 generic.go:356] "Generic (PLEG): container finished" podID="c1f3da8a-b765-44b0-b28d-c20f48cbf732" containerID="70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb" exitCode=0 Nov 02 08:37:59.853015 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.852888 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" Nov 02 08:37:59.853015 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.852933 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" event={"ID":"c1f3da8a-b765-44b0-b28d-c20f48cbf732","Type":"ContainerDied","Data":"70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb"} Nov 02 08:37:59.853015 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.852981 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk" event={"ID":"c1f3da8a-b765-44b0-b28d-c20f48cbf732","Type":"ContainerDied","Data":"261ff3e54d140a13d139b5e0fc5d58dab786177376a6857cbcf4629598545e8f"} Nov 02 08:37:59.862374 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.862156 6160 scope.go:122] "RemoveContainer" containerID="7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88" Nov 02 08:37:59.862519 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:37:59.862501 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88\": container with ID starting with 7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88 not found: ID does not exist" containerID="7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88" Nov 02 08:37:59.862567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.862528 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88"} err="failed to get container status \"7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88\": rpc error: code = NotFound desc = could not find container \"7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88\": container with ID starting with 7de93b3a67fcbe253dcc8168498afe858c24de4fcb994188b6f1c867f1b06a88 not found: ID does not exist" Nov 02 08:37:59.862567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.862544 6160 scope.go:122] "RemoveContainer" containerID="70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb" Nov 02 08:37:59.884328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.884304 6160 scope.go:122] "RemoveContainer" containerID="70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb" Nov 02 08:37:59.884649 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:37:59.884626 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb\": container with ID starting with 70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb not found: ID does not exist" containerID="70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb" Nov 02 08:37:59.884704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:37:59.884663 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb"} err="failed to get container status \"70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb\": rpc error: code = NotFound desc = could not find container \"70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb\": container with ID starting with 70970a2624878787dc3faebb3907c432958c3c2b9bb53f72d9dbbd266158d5eb not found: ID does not exist" Nov 02 08:38:00.005871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:00.005837 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk"] Nov 02 08:38:00.051878 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:00.051839 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6a289-7f9866786c-m9lpk"] Nov 02 08:38:00.136349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:00.136249 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769"] Nov 02 08:38:00.140199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:00.140156 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6a289-59d8578bc5-9t769"] Nov 02 08:38:00.702419 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:00.702389 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-pz258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:38:00.702841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:00.702440 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 08:38:01.585958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:01.585925 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570f3f22-3c1d-4206-b0dd-474023f4e48a" path="/var/lib/kubelet/pods/570f3f22-3c1d-4206-b0dd-474023f4e48a/volumes" Nov 02 08:38:01.586389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:01.586374 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f3da8a-b765-44b0-b28d-c20f48cbf732" path="/var/lib/kubelet/pods/c1f3da8a-b765-44b0-b28d-c20f48cbf732/volumes" Nov 02 08:38:01.702229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:01.702164 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-pz258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:38:01.702382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:01.702256 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 08:38:02.702333 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:02.702301 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-pz258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:38:02.702867 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:02.702369 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 08:38:03.164841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:03.164803 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:03.715362 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:03.715327 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:38:03.716298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:03.716276 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:38:03.717201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:03.717151 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:38:05.796408 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:05.796367 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:07.713337 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:07.713300 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:08.501407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:08.501372 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:08.678723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:08.678688 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:08.861007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:08.860904 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:10.514035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.513999 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm"] Nov 02 08:38:10.514935 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.514910 6160 cpu_manager.go:414] "containerMap: removing container" podUID="570f3f22-3c1d-4206-b0dd-474023f4e48a" containerName="payload-processing" Nov 02 08:38:10.515041 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.514935 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570f3f22-3c1d-4206-b0dd-474023f4e48a" containerName="payload-processing" Nov 02 08:38:10.515041 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.514963 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c1f3da8a-b765-44b0-b28d-c20f48cbf732" containerName="payload-pre-processing" Nov 02 08:38:10.515041 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.514971 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f3da8a-b765-44b0-b28d-c20f48cbf732" containerName="payload-pre-processing" Nov 02 08:38:10.515221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.515123 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="570f3f22-3c1d-4206-b0dd-474023f4e48a" containerName="payload-processing" Nov 02 08:38:10.515221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.515142 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f3da8a-b765-44b0-b28d-c20f48cbf732" containerName="payload-pre-processing" Nov 02 08:38:10.519336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.519311 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.528403 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:38:10.528367 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:38:10.628760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.628707 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm"] Nov 02 08:38:10.643546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.643507 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.643739 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.643569 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.643739 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.643642 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.643873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.643758 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.643873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.643834 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff836070-4c5d-4e76-bf8b-501c47770f50-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.643981 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.643885 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.643981 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.643915 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.643981 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.643947 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.644131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.643981 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fw6\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-kube-api-access-d7fw6\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746015 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.745981 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746066 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff836070-4c5d-4e76-bf8b-501c47770f50-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746102 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746119 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746150 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746319 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fw6\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-kube-api-access-d7fw6\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746481 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746645 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746535 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746645 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746611 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746645 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746620 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746798 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.746906 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746859 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.747030 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746931 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff836070-4c5d-4e76-bf8b-501c47770f50-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.747134 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.746980 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.748640 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.748616 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.748912 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.748893 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.801711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.801638 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.801890 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.801868 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fw6\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-kube-api-access-d7fw6\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-8shrm\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:10.835076 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:10.835048 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:11.255590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:11.255564 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm"] Nov 02 08:38:11.256669 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:38:11.256646 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff836070_4c5d_4e76_bf8b_501c47770f50.slice/crio-7dc3953a5ca578073d1555edd76400f9cd8cba2ac581fa05613eacb25e05e797 WatchSource:0}: Error finding container 7dc3953a5ca578073d1555edd76400f9cd8cba2ac581fa05613eacb25e05e797: Status 404 returned error can't find the container with id 7dc3953a5ca578073d1555edd76400f9cd8cba2ac581fa05613eacb25e05e797 Nov 02 08:38:11.259155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:11.259117 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:38:11.259270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:11.259222 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:38:11.259270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:11.259258 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:38:11.954414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:11.954376 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" event={"ID":"ff836070-4c5d-4e76-bf8b-501c47770f50","Type":"ContainerStarted","Data":"6ed9225c50c68e485845917cb0795a77b0f06f65ee2367376c9966f2ef484dcf"} Nov 02 08:38:11.954837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:11.954434 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" event={"ID":"ff836070-4c5d-4e76-bf8b-501c47770f50","Type":"ContainerStarted","Data":"7dc3953a5ca578073d1555edd76400f9cd8cba2ac581fa05613eacb25e05e797"} Nov 02 08:38:12.066215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:12.066132 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" podStartSLOduration=2.0661118 podStartE2EDuration="2.0661118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:12.063744707 +0000 UTC m=+1890.617248930" watchObservedRunningTime="2026-11-02 08:38:12.0661118 +0000 UTC m=+1890.619615999" Nov 02 08:38:12.194020 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:12.193969 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:12.403121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:12.403029 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:12.835535 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:12.835486 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:12.837698 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:12.837668 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 08:38:12.837857 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:12.837725 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 08:38:13.840214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:13.838220 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 08:38:13.840214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:13.838277 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 08:38:14.195128 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:14.195096 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:14.835718 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:14.835672 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 08:38:14.835902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:14.835740 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 08:38:15.836012 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:15.835978 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 08:38:15.836469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:15.836028 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 08:38:16.836200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:16.835934 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 08:38:16.836200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:16.836021 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 08:38:16.839304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:16.839278 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn"] Nov 02 08:38:16.839587 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:16.839557 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" containerID="cri-o://652b2b7569841174959b7f430d675e40ea854f5c60b705729d97ac76dfddd35f" gracePeriod=30 Nov 02 08:38:18.282156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:18.282116 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:18.282714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:18.282463 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:18.283551 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:18.283523 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:38:19.738064 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:19.738025 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:20.894050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:20.894014 6160 patch_prober.go:29] interesting pod/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:38:20.894459 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:20.894069 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:38:22.035136 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.035009 6160 generic.go:356] "Generic (PLEG): container finished" podID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerID="652b2b7569841174959b7f430d675e40ea854f5c60b705729d97ac76dfddd35f" exitCode=0 Nov 02 08:38:22.035620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.035231 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" event={"ID":"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7","Type":"ContainerDied","Data":"652b2b7569841174959b7f430d675e40ea854f5c60b705729d97ac76dfddd35f"} Nov 02 08:38:22.117739 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.117670 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:38:22.231784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.231752 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-socket\") pod \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " Nov 02 08:38:22.231989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.231792 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-certs\") pod \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " Nov 02 08:38:22.231989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.231863 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-token\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-token\") pod \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " Nov 02 08:38:22.231989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.231916 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-kube-api-access-jqp62\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-kube-api-access-jqp62\") pod \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " Nov 02 08:38:22.232158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.231994 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-credential-socket\") pod \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " Nov 02 08:38:22.232158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.232027 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-data\") pod \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " Nov 02 08:38:22.232158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.232068 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istiod-ca-cert\") pod \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " Nov 02 08:38:22.232158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.232102 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-podinfo\") pod \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " Nov 02 08:38:22.232158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.232108 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-socket" pod "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" (UID: "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:22.232460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.232299 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-envoy\") pod \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\" (UID: \"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7\") " Nov 02 08:38:22.232460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.232315 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-certs" pod "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" (UID: "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:22.232460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.232401 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-data" pod "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" (UID: "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:22.232615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.232457 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-credential-socket" pod "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" (UID: "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:22.232615 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.232495 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istiod-ca-cert" pod "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" (UID: "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:22.233224 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.233164 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:22.233224 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.233226 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:22.233441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.233240 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:22.233441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.233252 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:22.233441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.233264 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:22.234691 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.234663 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-token" pod "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" (UID: "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:22.234805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.234707 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-podinfo" pod "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" (UID: "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:38:22.235109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.235085 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-envoy" pod "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" (UID: "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:22.235109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.235091 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-kube-api-access-jqp62" pod "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" (UID: "fab5f20f-4fc2-44bd-9d08-dfce7903b6d7"). InnerVolumeSpecName "kube-api-access-jqp62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:22.334956 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.334915 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:22.334956 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.334948 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:22.334956 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.334962 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqp62\" (UniqueName: \"kubernetes.io/projected/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-kube-api-access-jqp62\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:22.335241 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:22.334989 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:23.054245 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:23.054205 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" Nov 02 08:38:23.054738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:23.054207 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn" event={"ID":"fab5f20f-4fc2-44bd-9d08-dfce7903b6d7","Type":"ContainerDied","Data":"a0a588ec97fa59bb44bcef6ba8644350217da945031cadd1b1af6266b46a9d16"} Nov 02 08:38:23.054738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:23.054385 6160 scope.go:122] "RemoveContainer" containerID="652b2b7569841174959b7f430d675e40ea854f5c60b705729d97ac76dfddd35f" Nov 02 08:38:23.202386 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:23.202357 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:23.257305 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:23.257272 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn"] Nov 02 08:38:23.302920 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:23.302884 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9d77b6-openshift-default-5c9c5679b5-f7kvn"] Nov 02 08:38:23.425419 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:23.425387 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:23.585159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:23.585124 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" path="/var/lib/kubelet/pods/fab5f20f-4fc2-44bd-9d08-dfce7903b6d7/volumes" Nov 02 08:38:23.609468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:23.609422 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:24.186983 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.186946 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg"] Nov 02 08:38:24.187425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.187167 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" podUID="43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" containerName="maas-api" containerID="cri-o://4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477" gracePeriod=30 Nov 02 08:38:24.318210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.318136 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:24.485456 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.485419 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:38:24.569490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.569454 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls\") pod \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " Nov 02 08:38:24.569678 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.569598 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-metrics-certs\") pod \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " Nov 02 08:38:24.569678 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.569642 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-openshift-service-ca\") pod \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " Nov 02 08:38:24.569762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.569708 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-kube-api-access-fgfl2\" (UniqueName: \"kubernetes.io/projected/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-kube-api-access-fgfl2\") pod \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\" (UID: \"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a\") " Nov 02 08:38:24.570049 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.570022 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-openshift-service-ca" pod "43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" (UID: "43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:24.570249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.570230 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:24.572121 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.572093 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls" pod "43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" (UID: "43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:38:24.572233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.572142 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-metrics-certs" pod "43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" (UID: "43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:38:24.572279 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.572246 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-kube-api-access-fgfl2" pod "43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" (UID: "43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a"). InnerVolumeSpecName "kube-api-access-fgfl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:24.677520 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.677479 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:24.677520 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.677520 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgfl2\" (UniqueName: \"kubernetes.io/projected/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-kube-api-access-fgfl2\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:24.677803 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:24.677537 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:38:25.071301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.071267 6160 generic.go:356] "Generic (PLEG): container finished" podID="43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" containerID="4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477" exitCode=0 Nov 02 08:38:25.071515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.071325 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" Nov 02 08:38:25.071515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.071349 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" event={"ID":"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a","Type":"ContainerDied","Data":"4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477"} Nov 02 08:38:25.071515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.071402 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg" event={"ID":"43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a","Type":"ContainerDied","Data":"1cbc0463af3e7a89ec4798ea2aabb5b80e133d0701e7bc35c4e61075410d5a88"} Nov 02 08:38:25.071515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.071424 6160 scope.go:122] "RemoveContainer" containerID="4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477" Nov 02 08:38:25.087441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.087414 6160 scope.go:122] "RemoveContainer" containerID="4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477" Nov 02 08:38:25.087932 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:38:25.087910 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477\": container with ID starting with 4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477 not found: ID does not exist" containerID="4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477" Nov 02 08:38:25.088030 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.087939 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477"} err="failed to get container status \"4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477\": rpc error: code = NotFound desc = could not find container \"4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477\": container with ID starting with 4a833b4223b01ec8d79edf5ef88a93c865c8bb62732bd43026d6f3b2c2aaa477 not found: ID does not exist" Nov 02 08:38:25.237371 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.237333 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg"] Nov 02 08:38:25.280931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.280895 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00cc0e-69d5865944-xqwjg"] Nov 02 08:38:25.585533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.585502 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" path="/var/lib/kubelet/pods/43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a/volumes" Nov 02 08:38:25.615681 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.615652 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4"] Nov 02 08:38:25.616300 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.616286 6160 cpu_manager.go:414] "containerMap: removing container" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" Nov 02 08:38:25.616352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.616301 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" Nov 02 08:38:25.616352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.616338 6160 cpu_manager.go:414] "containerMap: removing container" podUID="43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" containerName="maas-api" Nov 02 08:38:25.616352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.616343 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" containerName="maas-api" Nov 02 08:38:25.616451 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.616433 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab5f20f-4fc2-44bd-9d08-dfce7903b6d7" containerName="istio-proxy" Nov 02 08:38:25.616451 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.616447 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c8a1b4-cf9e-45c0-a092-c64a2ece9c4a" containerName="maas-api" Nov 02 08:38:25.621338 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.621321 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.677608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.677580 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4"] Nov 02 08:38:25.711406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.711372 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.711576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.711425 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.711620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.711564 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.711694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.711676 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kf2t\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-kube-api-access-8kf2t\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.711744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.711732 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.711789 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.711777 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.711825 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.711797 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.711861 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.711823 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bba8a35-6019-4aa7-861d-2bb576831fef-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.711966 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.711944 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813057 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kf2t\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-kube-api-access-8kf2t\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813102 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813143 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813160 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813212 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bba8a35-6019-4aa7-861d-2bb576831fef-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813258 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813583 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813402 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813583 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813460 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813583 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813527 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813650 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-data\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813859 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-credential-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.813959 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.813937 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bba8a35-6019-4aa7-861d-2bb576831fef-istiod-ca-cert\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.814041 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.814021 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-socket\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.814078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.814026 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-certs\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.815866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.815845 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-envoy\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.816057 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.816039 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-podinfo\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.866515 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.866448 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-token\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.866688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.866670 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kf2t\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-kube-api-access-8kf2t\") pod \"e2e-mt-cba6f9af-openshift-default-db985b657-24dg4\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:25.935169 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:25.935135 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:26.249396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:26.249355 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:26.355735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:26.355699 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4"] Nov 02 08:38:26.355979 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:38:26.355960 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bba8a35_6019_4aa7_861d_2bb576831fef.slice/crio-aa1841358e6c17856d79e6d7551b43fc7c056369238062e6d0fceb821eec2856 WatchSource:0}: Error finding container aa1841358e6c17856d79e6d7551b43fc7c056369238062e6d0fceb821eec2856: Status 404 returned error can't find the container with id aa1841358e6c17856d79e6d7551b43fc7c056369238062e6d0fceb821eec2856 Nov 02 08:38:26.358622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:26.358589 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:38:26.358696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:26.358653 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:38:26.358696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:26.358684 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:38:26.510034 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:26.509934 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:26.703827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:26.703794 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:27.019019 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.018980 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:27.059034 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.058985 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4"] Nov 02 08:38:27.084237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.065539 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:27.114043 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.114000 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" event={"ID":"0bba8a35-6019-4aa7-861d-2bb576831fef","Type":"ContainerStarted","Data":"991ac18cb46aed4599ed03c2188ee57c14a2d010554684a821b792b9b2abbe02"} Nov 02 08:38:27.114321 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.114301 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" event={"ID":"0bba8a35-6019-4aa7-861d-2bb576831fef","Type":"ContainerStarted","Data":"aa1841358e6c17856d79e6d7551b43fc7c056369238062e6d0fceb821eec2856"} Nov 02 08:38:27.123781 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.122921 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4"] Nov 02 08:38:27.140532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.140496 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69146e90-5eae-4285-83b3-146083c3e96f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4\" (UID: \"69146e90-5eae-4285-83b3-146083c3e96f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:27.140707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.140539 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqmx\" (UniqueName: \"kubernetes.io/projected/69146e90-5eae-4285-83b3-146083c3e96f-kube-api-access-jvqmx\") pod \"payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4\" (UID: \"69146e90-5eae-4285-83b3-146083c3e96f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:27.159092 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.159057 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f4516e-dockercfg-p7k4s\"" Nov 02 08:38:27.163244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.163217 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f4516e\"" Nov 02 08:38:27.241497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.241454 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69146e90-5eae-4285-83b3-146083c3e96f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4\" (UID: \"69146e90-5eae-4285-83b3-146083c3e96f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:27.241687 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.241507 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqmx\" (UniqueName: \"kubernetes.io/projected/69146e90-5eae-4285-83b3-146083c3e96f-kube-api-access-jvqmx\") pod \"payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4\" (UID: \"69146e90-5eae-4285-83b3-146083c3e96f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:27.242354 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.242326 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69146e90-5eae-4285-83b3-146083c3e96f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4\" (UID: \"69146e90-5eae-4285-83b3-146083c3e96f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:27.295413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.295267 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" podStartSLOduration=2.295251137 podStartE2EDuration="2.295251137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:27.291546103 +0000 UTC m=+1905.845050299" watchObservedRunningTime="2026-11-02 08:38:27.295251137 +0000 UTC m=+1905.848755334" Nov 02 08:38:27.299260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.299231 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqmx\" (UniqueName: \"kubernetes.io/projected/69146e90-5eae-4285-83b3-146083c3e96f-kube-api-access-jvqmx\") pod \"payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4\" (UID: \"69146e90-5eae-4285-83b3-146083c3e96f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:27.300018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.299989 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:27.442370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.442336 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:27.836236 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:38:27.836197 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69146e90_5eae_4285_83b3_146083c3e96f.slice/crio-bc4bb2026f7d3187a53d227712c702a3c9b2d326c8dedc7a5d821602443d521c WatchSource:0}: Error finding container bc4bb2026f7d3187a53d227712c702a3c9b2d326c8dedc7a5d821602443d521c: Status 404 returned error can't find the container with id bc4bb2026f7d3187a53d227712c702a3c9b2d326c8dedc7a5d821602443d521c Nov 02 08:38:27.837516 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.837491 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4"] Nov 02 08:38:27.936112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.936083 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:27.937757 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.937729 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 08:38:27.937899 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.937785 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 08:38:27.973625 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:27.973600 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:28.139431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:28.139318 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" event={"ID":"69146e90-5eae-4285-83b3-146083c3e96f","Type":"ContainerStarted","Data":"f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a"} Nov 02 08:38:28.139431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:28.139388 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" event={"ID":"69146e90-5eae-4285-83b3-146083c3e96f","Type":"ContainerStarted","Data":"bc4bb2026f7d3187a53d227712c702a3c9b2d326c8dedc7a5d821602443d521c"} Nov 02 08:38:28.139431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:28.139427 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:28.249309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:28.249248 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" podStartSLOduration=1.249228354 podStartE2EDuration="1.249228354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:28.246046712 +0000 UTC m=+1906.799550909" watchObservedRunningTime="2026-11-02 08:38:28.249228354 +0000 UTC m=+1906.802732555" Nov 02 08:38:28.936162 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:28.936124 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 08:38:28.936590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:28.936196 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 08:38:29.935722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:29.935683 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 08:38:29.935903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:29.935740 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 08:38:30.936410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:30.936371 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 08:38:30.936880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:30.936443 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 08:38:31.540481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:31.540439 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:31.935690 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:31.935654 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 08:38:31.935881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:31.935710 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 08:38:32.936111 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:32.936064 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 08:38:32.936725 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:32.936121 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 08:38:33.940160 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:33.940127 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:33.940665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:33.940601 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:33.941425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:33.941393 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:38:33.964887 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:33.964859 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:34.187042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:34.187006 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:34.430879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:34.430839 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:36.952758 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:36.952717 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:39.149352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:39.149312 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:38:40.522929 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:38:40.522885 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:38:43.406620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:43.406570 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:48.133401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.133354 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:48.283413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.283374 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq"] Nov 02 08:38:48.290139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.290115 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.339054 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.339023 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d51175-serving-cert\"" Nov 02 08:38:48.383315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.383259 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b483ec-087c-4c10-973c-fea9a66a1073-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.383495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.383398 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.383495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.383473 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-metrics-certs\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.383618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.383555 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kfhr\" (UniqueName: \"kubernetes.io/projected/96b483ec-087c-4c10-973c-fea9a66a1073-kube-api-access-8kfhr\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.389600 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.389570 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq"] Nov 02 08:38:48.485300 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.485264 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-metrics-certs\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.485475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.485313 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kfhr\" (UniqueName: \"kubernetes.io/projected/96b483ec-087c-4c10-973c-fea9a66a1073-kube-api-access-8kfhr\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.485543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.485490 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b483ec-087c-4c10-973c-fea9a66a1073-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.485543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.485527 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.485666 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:38:48.485624 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-serving-cert: secret "maas-api-e2e-ipp-a-d51175-serving-cert" not found Nov 02 08:38:48.485731 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:38:48.485715 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls podName:96b483ec-087c-4c10-973c-fea9a66a1073 nodeName:}" failed. No retries permitted until 2026-11-02 08:38:48.985693834 +0000 UTC m=+1927.539198026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls") pod "maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" (UID: "96b483ec-087c-4c10-973c-fea9a66a1073") : secret "maas-api-e2e-ipp-a-d51175-serving-cert" not found Nov 02 08:38:48.486077 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.486057 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b483ec-087c-4c10-973c-fea9a66a1073-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.487966 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.487948 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-metrics-certs\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.514221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.514150 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:48.539498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.539468 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kfhr\" (UniqueName: \"kubernetes.io/projected/96b483ec-087c-4c10-973c-fea9a66a1073-kube-api-access-8kfhr\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.685196 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.685149 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:48.854109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.854059 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:48.991856 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.991761 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:48.994380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:48.994359 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:49.205128 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:49.205096 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:49.579194 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:38:49.579150 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b483ec_087c_4c10_973c_fea9a66a1073.slice/crio-eaba2ad8bf6af85488df31a418c48ae1083f77ea706c5c3cfed7939919e1c831 WatchSource:0}: Error finding container eaba2ad8bf6af85488df31a418c48ae1083f77ea706c5c3cfed7939919e1c831: Status 404 returned error can't find the container with id eaba2ad8bf6af85488df31a418c48ae1083f77ea706c5c3cfed7939919e1c831 Nov 02 08:38:49.587742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:49.587719 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq"] Nov 02 08:38:50.305166 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:50.305130 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" event={"ID":"96b483ec-087c-4c10-973c-fea9a66a1073","Type":"ContainerStarted","Data":"eaba2ad8bf6af85488df31a418c48ae1083f77ea706c5c3cfed7939919e1c831"} Nov 02 08:38:51.314191 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:51.314148 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" event={"ID":"96b483ec-087c-4c10-973c-fea9a66a1073","Type":"ContainerStarted","Data":"bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8"} Nov 02 08:38:51.314677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:51.314308 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:51.464064 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:51.464006 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" podStartSLOduration=2.494824289 podStartE2EDuration="3.463992038s" totalImagesPullingTime="969.167749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:51.459879845 +0000 UTC m=+1930.013384042" watchObservedRunningTime="2026-11-02 08:38:51.463992038 +0000 UTC m=+1930.017496234" Nov 02 08:38:53.951430 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:53.951388 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:54.608703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:54.608666 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:54.896566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:54.896485 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:56.173407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:56.173367 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:56.334714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:56.334677 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:56.686377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:56.686336 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:57.341830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:57.341802 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:38:58.672623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:38:58.671738 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:00.558223 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:00.558188 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:00.724654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:00.724612 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:00.921282 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:00.921240 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:03.719726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:03.719694 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9"] Nov 02 08:39:03.720145 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:03.719894 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" containerID="cri-o://edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f" gracePeriod=30 Nov 02 08:39:05.333090 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:05.333049 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:05.500995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:05.500945 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:05.504796 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:05.504762 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:39:05.504943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:05.504830 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:39:05.669247 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:05.669211 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:06.896272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:06.896227 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:07.075992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:07.075955 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:07.237137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:07.237102 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:07.406774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:07.406740 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:08.996149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:08.996075 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:39:09.155918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.155879 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-data\") pod \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " Nov 02 08:39:09.156108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.155945 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-envoy\") pod \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " Nov 02 08:39:09.156108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.156021 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-podinfo\") pod \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " Nov 02 08:39:09.156108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.156066 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-token\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-token\") pod \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " Nov 02 08:39:09.156325 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.156107 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-certs\") pod \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " Nov 02 08:39:09.156325 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.156143 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istiod-ca-cert\") pod \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " Nov 02 08:39:09.156325 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.156202 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-kube-api-access-2dz2g\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-kube-api-access-2dz2g\") pod \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " Nov 02 08:39:09.156480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.156393 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-credential-socket\") pod \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " Nov 02 08:39:09.156480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.156446 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-socket\") pod \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\" (UID: \"2c7a06ed-0628-42db-9682-2e3ec8e5bca7\") " Nov 02 08:39:09.156594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.156529 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-data" pod "2c7a06ed-0628-42db-9682-2e3ec8e5bca7" (UID: "2c7a06ed-0628-42db-9682-2e3ec8e5bca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:09.157114 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.157090 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:09.157543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.157521 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-socket" pod "2c7a06ed-0628-42db-9682-2e3ec8e5bca7" (UID: "2c7a06ed-0628-42db-9682-2e3ec8e5bca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:09.158755 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.157563 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-certs" pod "2c7a06ed-0628-42db-9682-2e3ec8e5bca7" (UID: "2c7a06ed-0628-42db-9682-2e3ec8e5bca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:09.158834 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.157589 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istiod-ca-cert" pod "2c7a06ed-0628-42db-9682-2e3ec8e5bca7" (UID: "2c7a06ed-0628-42db-9682-2e3ec8e5bca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:09.158889 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.158869 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-credential-socket" pod "2c7a06ed-0628-42db-9682-2e3ec8e5bca7" (UID: "2c7a06ed-0628-42db-9682-2e3ec8e5bca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:09.159636 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.159470 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-podinfo" pod "2c7a06ed-0628-42db-9682-2e3ec8e5bca7" (UID: "2c7a06ed-0628-42db-9682-2e3ec8e5bca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:09.159636 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.159579 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-envoy" pod "2c7a06ed-0628-42db-9682-2e3ec8e5bca7" (UID: "2c7a06ed-0628-42db-9682-2e3ec8e5bca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:09.160235 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.160156 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-kube-api-access-2dz2g" pod "2c7a06ed-0628-42db-9682-2e3ec8e5bca7" (UID: "2c7a06ed-0628-42db-9682-2e3ec8e5bca7"). InnerVolumeSpecName "kube-api-access-2dz2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:09.161283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.161259 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-token" pod "2c7a06ed-0628-42db-9682-2e3ec8e5bca7" (UID: "2c7a06ed-0628-42db-9682-2e3ec8e5bca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:09.259524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.259429 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:09.259524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.259468 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:09.259524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.259483 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:09.259524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.259497 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:09.259524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.259511 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:09.259524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.259524 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:09.259848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.259538 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:09.259848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.259551 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dz2g\" (UniqueName: \"kubernetes.io/projected/2c7a06ed-0628-42db-9682-2e3ec8e5bca7-kube-api-access-2dz2g\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:09.487495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.487462 6160 generic.go:356] "Generic (PLEG): container finished" podID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerID="edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f" exitCode=0 Nov 02 08:39:09.487714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.487523 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" Nov 02 08:39:09.487714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.487551 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" event={"ID":"2c7a06ed-0628-42db-9682-2e3ec8e5bca7","Type":"ContainerDied","Data":"edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f"} Nov 02 08:39:09.487714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.487604 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9" event={"ID":"2c7a06ed-0628-42db-9682-2e3ec8e5bca7","Type":"ContainerDied","Data":"4d8ee6adeae251317c80e89711b6a26c89df13c45633ba1eb97ba2db5570eb51"} Nov 02 08:39:09.487714 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.487628 6160 scope.go:122] "RemoveContainer" containerID="edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f" Nov 02 08:39:09.499897 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.499875 6160 scope.go:122] "RemoveContainer" containerID="edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f" Nov 02 08:39:09.500252 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:39:09.500225 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f\": container with ID starting with edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f not found: ID does not exist" containerID="edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f" Nov 02 08:39:09.500360 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.500265 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f"} err="failed to get container status \"edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f\": rpc error: code = NotFound desc = could not find container \"edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f\": container with ID starting with edce21da1995cd4e04f6eadd7cdb23ae3133d68645264d22583a0c6f16919e9f not found: ID does not exist" Nov 02 08:39:09.645263 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.645147 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9"] Nov 02 08:39:09.691008 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:09.690976 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-c7bj9"] Nov 02 08:39:10.522243 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:39:10.522205 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:39:11.508920 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:11.508889 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:11.594582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:11.594548 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" path="/var/lib/kubelet/pods/2c7a06ed-0628-42db-9682-2e3ec8e5bca7/volumes" Nov 02 08:39:11.709956 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:11.709919 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb"] Nov 02 08:39:11.710135 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:11.710106 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" podUID="71436aad-0a0c-4d55-9f18-3c147206e3b0" containerName="maas-api" containerID="cri-o://17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8" gracePeriod=30 Nov 02 08:39:11.844373 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:11.844327 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:12.006792 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.006761 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:39:12.097746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.097652 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls\") pod \"71436aad-0a0c-4d55-9f18-3c147206e3b0\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " Nov 02 08:39:12.097933 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.097752 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71436aad-0a0c-4d55-9f18-3c147206e3b0-kube-api-access-dlm69\" (UniqueName: \"kubernetes.io/projected/71436aad-0a0c-4d55-9f18-3c147206e3b0-kube-api-access-dlm69\") pod \"71436aad-0a0c-4d55-9f18-3c147206e3b0\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " Nov 02 08:39:12.097933 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.097802 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-metrics-certs\") pod \"71436aad-0a0c-4d55-9f18-3c147206e3b0\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " Nov 02 08:39:12.098051 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.097983 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71436aad-0a0c-4d55-9f18-3c147206e3b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71436aad-0a0c-4d55-9f18-3c147206e3b0-openshift-service-ca\") pod \"71436aad-0a0c-4d55-9f18-3c147206e3b0\" (UID: \"71436aad-0a0c-4d55-9f18-3c147206e3b0\") " Nov 02 08:39:12.098347 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.098320 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71436aad-0a0c-4d55-9f18-3c147206e3b0-openshift-service-ca" pod "71436aad-0a0c-4d55-9f18-3c147206e3b0" (UID: "71436aad-0a0c-4d55-9f18-3c147206e3b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:12.098561 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.098543 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71436aad-0a0c-4d55-9f18-3c147206e3b0-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:12.100314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.100289 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71436aad-0a0c-4d55-9f18-3c147206e3b0-kube-api-access-dlm69" pod "71436aad-0a0c-4d55-9f18-3c147206e3b0" (UID: "71436aad-0a0c-4d55-9f18-3c147206e3b0"). InnerVolumeSpecName "kube-api-access-dlm69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:12.100592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.100572 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls" pod "71436aad-0a0c-4d55-9f18-3c147206e3b0" (UID: "71436aad-0a0c-4d55-9f18-3c147206e3b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:39:12.100672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.100647 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-metrics-certs" pod "71436aad-0a0c-4d55-9f18-3c147206e3b0" (UID: "71436aad-0a0c-4d55-9f18-3c147206e3b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:39:12.200005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.199970 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:12.200005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.200003 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlm69\" (UniqueName: \"kubernetes.io/projected/71436aad-0a0c-4d55-9f18-3c147206e3b0-kube-api-access-dlm69\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:12.200005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.200014 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71436aad-0a0c-4d55-9f18-3c147206e3b0-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:12.513127 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.513089 6160 generic.go:356] "Generic (PLEG): container finished" podID="71436aad-0a0c-4d55-9f18-3c147206e3b0" containerID="17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8" exitCode=0 Nov 02 08:39:12.513356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.513132 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" Nov 02 08:39:12.513356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.513134 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" event={"ID":"71436aad-0a0c-4d55-9f18-3c147206e3b0","Type":"ContainerDied","Data":"17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8"} Nov 02 08:39:12.513356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.513217 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb" event={"ID":"71436aad-0a0c-4d55-9f18-3c147206e3b0","Type":"ContainerDied","Data":"ac760468a0048f1ce0ea84e1f7af6cb3a144507a1b028821ab185934b5629263"} Nov 02 08:39:12.513356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.513250 6160 scope.go:122] "RemoveContainer" containerID="17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8" Nov 02 08:39:12.525133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.525115 6160 scope.go:122] "RemoveContainer" containerID="17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8" Nov 02 08:39:12.525473 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:39:12.525448 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8\": container with ID starting with 17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8 not found: ID does not exist" containerID="17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8" Nov 02 08:39:12.525534 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.525483 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8"} err="failed to get container status \"17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8\": rpc error: code = NotFound desc = could not find container \"17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8\": container with ID starting with 17506fe4d1f17bdafc795002e5985aad24753fd11ee55d7d2b049b26b02e39f8 not found: ID does not exist" Nov 02 08:39:12.669229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.669163 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb"] Nov 02 08:39:12.673133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:12.673106 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7f7d7-f86766578-bwndb"] Nov 02 08:39:13.512975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:13.512936 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:13.586727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:13.586694 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71436aad-0a0c-4d55-9f18-3c147206e3b0" path="/var/lib/kubelet/pods/71436aad-0a0c-4d55-9f18-3c147206e3b0/volumes" Nov 02 08:39:13.726882 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:13.726842 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:13.911839 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:13.911804 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:14.406729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:14.406640 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:16.531732 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:16.531683 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:17.873289 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:17.873249 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:18.959661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:18.959610 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:20.103142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:20.103101 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:20.317360 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:20.317328 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:20.495252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:20.495220 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:22.267664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:22.267629 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:22.430602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:22.430561 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:23.562201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:23.557142 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:23.859406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:23.859299 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:24.135666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:24.135565 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:24.319307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:24.319265 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:24.542370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:24.542329 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:24.970892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:24.970858 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:29.302577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:29.302539 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:29.904397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:29.904362 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44"] Nov 02 08:39:30.046918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:30.046882 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:31.393192 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:31.393133 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:32.019291 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:32.019251 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:32.515797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:32.515757 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:34.065504 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.065464 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9"] Nov 02 08:39:34.066377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.066354 6160 cpu_manager.go:414] "containerMap: removing container" podUID="71436aad-0a0c-4d55-9f18-3c147206e3b0" containerName="maas-api" Nov 02 08:39:34.066377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.066377 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71436aad-0a0c-4d55-9f18-3c147206e3b0" containerName="maas-api" Nov 02 08:39:34.066550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.066415 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" Nov 02 08:39:34.066550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.066424 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" Nov 02 08:39:34.066653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.066579 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7a06ed-0628-42db-9682-2e3ec8e5bca7" containerName="istio-proxy" Nov 02 08:39:34.066653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.066606 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="71436aad-0a0c-4d55-9f18-3c147206e3b0" containerName="maas-api" Nov 02 08:39:34.070263 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.070243 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.120119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.120088 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f4516e-serving-cert\"" Nov 02 08:39:34.132733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.132695 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmjh\" (UniqueName: \"kubernetes.io/projected/514dee5e-6142-441c-8bba-95bab2a0be1c-kube-api-access-qhmjh\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.132927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.132765 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-metrics-certs\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.133001 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.132961 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.133124 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.133106 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514dee5e-6142-441c-8bba-95bab2a0be1c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.170595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.170560 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9"] Nov 02 08:39:34.234430 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.234398 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.234620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.234524 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514dee5e-6142-441c-8bba-95bab2a0be1c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.234620 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:39:34.234568 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-serving-cert: secret "maas-api-e2e-praxis-a-f4516e-serving-cert" not found Nov 02 08:39:34.234721 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:39:34.234636 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls podName:514dee5e-6142-441c-8bba-95bab2a0be1c nodeName:}" failed. No retries permitted until 2026-11-02 08:39:34.734615589 +0000 UTC m=+1973.288119781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls") pod "maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" (UID: "514dee5e-6142-441c-8bba-95bab2a0be1c") : secret "maas-api-e2e-praxis-a-f4516e-serving-cert" not found Nov 02 08:39:34.234721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.234576 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmjh\" (UniqueName: \"kubernetes.io/projected/514dee5e-6142-441c-8bba-95bab2a0be1c-kube-api-access-qhmjh\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.234838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.234750 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-metrics-certs\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.235339 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.235315 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514dee5e-6142-441c-8bba-95bab2a0be1c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.237392 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.237369 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-metrics-certs\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.288608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.288571 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmjh\" (UniqueName: \"kubernetes.io/projected/514dee5e-6142-441c-8bba-95bab2a0be1c-kube-api-access-qhmjh\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.305762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.305730 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:34.492829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.492789 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:34.668955 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.668920 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:34.745377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.745285 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.747874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.747852 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:34.983829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:34.983792 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:35.316592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:35.316504 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:35.366005 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:39:35.365975 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514dee5e_6142_441c_8bba_95bab2a0be1c.slice/crio-b0e1a86fb2865433c49adc2c6b4350edd71d059022a453f52e79a369ae526ca5 WatchSource:0}: Error finding container b0e1a86fb2865433c49adc2c6b4350edd71d059022a453f52e79a369ae526ca5: Status 404 returned error can't find the container with id b0e1a86fb2865433c49adc2c6b4350edd71d059022a453f52e79a369ae526ca5 Nov 02 08:39:35.366186 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:35.366151 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9"] Nov 02 08:39:35.696096 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:35.696062 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" event={"ID":"514dee5e-6142-441c-8bba-95bab2a0be1c","Type":"ContainerStarted","Data":"b0e1a86fb2865433c49adc2c6b4350edd71d059022a453f52e79a369ae526ca5"} Nov 02 08:39:36.234022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:36.233983 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg"] Nov 02 08:39:36.345360 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:36.345325 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86"] Nov 02 08:39:36.648000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:36.647915 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:36.709858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:36.709812 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" event={"ID":"514dee5e-6142-441c-8bba-95bab2a0be1c","Type":"ContainerStarted","Data":"33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19"} Nov 02 08:39:36.710393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:36.710358 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:36.819017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:36.818967 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:36.824969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:36.824911 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" podStartSLOduration=1.935388982 podStartE2EDuration="2.824896261s" totalImagesPullingTime="889.507279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:36.821483412 +0000 UTC m=+1975.374987627" watchObservedRunningTime="2026-11-02 08:39:36.824896261 +0000 UTC m=+1975.378400452" Nov 02 08:39:37.025324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:37.025281 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:37.198829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:37.198791 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:39.418117 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.418081 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8"] Nov 02 08:39:39.418596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.418327 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" podUID="c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" containerName="maas-api" containerID="cri-o://c6ef9174351e55a15dc58d7c2fc59ca1be2e5bb82d226c457366dd978995ee2d" gracePeriod=30 Nov 02 08:39:39.518880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.518843 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc"] Nov 02 08:39:39.519156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.519123 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" containerID="cri-o://7c5cf1fc5ed17591a21c210f7ec58036b7a135db544745b8da918837948fc088" gracePeriod=30 Nov 02 08:39:39.523158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.523131 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4"] Nov 02 08:39:39.523444 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.523413 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" containerID="cri-o://a31639e9114b7ca946d277529f9c7cbcf8f57de9cf71a29d28fa04204df59ee9" gracePeriod=30 Nov 02 08:39:39.561128 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.561091 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:39.755367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.755328 6160 generic.go:356] "Generic (PLEG): container finished" podID="c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" containerID="c6ef9174351e55a15dc58d7c2fc59ca1be2e5bb82d226c457366dd978995ee2d" exitCode=0 Nov 02 08:39:39.755654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.755634 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" event={"ID":"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc","Type":"ContainerDied","Data":"c6ef9174351e55a15dc58d7c2fc59ca1be2e5bb82d226c457366dd978995ee2d"} Nov 02 08:39:39.755729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.755693 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" event={"ID":"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc","Type":"ContainerDied","Data":"c8905d4b95eab104f70900f530f1e07286af34eb32f2983fde523e711ced739b"} Nov 02 08:39:39.755729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.755719 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8905d4b95eab104f70900f530f1e07286af34eb32f2983fde523e711ced739b" Nov 02 08:39:39.760066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.760034 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:39:39.828103 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.828073 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-kube-api-access-hsv89\" (UniqueName: \"kubernetes.io/projected/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-kube-api-access-hsv89\") pod \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " Nov 02 08:39:39.828315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.828121 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-metrics-certs\") pod \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " Nov 02 08:39:39.828315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.828260 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls\") pod \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " Nov 02 08:39:39.828315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.828303 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-openshift-service-ca\") pod \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\" (UID: \"c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc\") " Nov 02 08:39:39.828760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.828730 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-openshift-service-ca" pod "c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" (UID: "c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:39.829065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.828974 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:39.830439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.830417 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-metrics-certs" pod "c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" (UID: "c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:39:39.830542 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.830529 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls" pod "c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" (UID: "c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:39:39.830542 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.830529 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-kube-api-access-hsv89" pod "c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" (UID: "c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc"). InnerVolumeSpecName "kube-api-access-hsv89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:39.931042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.931005 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsv89\" (UniqueName: \"kubernetes.io/projected/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-kube-api-access-hsv89\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:39.931042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.931045 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:39.931324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:39.931059 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:40.524537 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:39:40.524495 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:39:40.762496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:40.762464 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8" Nov 02 08:39:40.933832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:40.933800 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8"] Nov 02 08:39:40.980284 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:40.980247 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9eea3-7d5fbdcc7c-97nd8"] Nov 02 08:39:41.235158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.235053 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" podUID="f9887717-ef97-4158-95c7-dc9149c57caf" containerName="payload-processing" containerID="cri-o://390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271" gracePeriod=25 Nov 02 08:39:41.346074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.346025 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" podUID="ca996665-95bf-48ec-982c-73306576f87e" containerName="payload-pre-processing" containerID="cri-o://28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6" gracePeriod=25 Nov 02 08:39:41.519391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.519364 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:39:41.588845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.588809 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" path="/var/lib/kubelet/pods/c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc/volumes" Nov 02 08:39:41.638272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.638246 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:39:41.658444 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.658415 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9887717-ef97-4158-95c7-dc9149c57caf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9887717-ef97-4158-95c7-dc9149c57caf-plugins-config-volume\") pod \"f9887717-ef97-4158-95c7-dc9149c57caf\" (UID: \"f9887717-ef97-4158-95c7-dc9149c57caf\") " Nov 02 08:39:41.658622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.658475 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9887717-ef97-4158-95c7-dc9149c57caf-kube-api-access-zwmlm\" (UniqueName: \"kubernetes.io/projected/f9887717-ef97-4158-95c7-dc9149c57caf-kube-api-access-zwmlm\") pod \"f9887717-ef97-4158-95c7-dc9149c57caf\" (UID: \"f9887717-ef97-4158-95c7-dc9149c57caf\") " Nov 02 08:39:41.658771 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.658747 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9887717-ef97-4158-95c7-dc9149c57caf-plugins-config-volume" pod "f9887717-ef97-4158-95c7-dc9149c57caf" (UID: "f9887717-ef97-4158-95c7-dc9149c57caf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:41.659285 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.659167 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9887717-ef97-4158-95c7-dc9149c57caf-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:41.661107 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.661081 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9887717-ef97-4158-95c7-dc9149c57caf-kube-api-access-zwmlm" pod "f9887717-ef97-4158-95c7-dc9149c57caf" (UID: "f9887717-ef97-4158-95c7-dc9149c57caf"). InnerVolumeSpecName "kube-api-access-zwmlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:41.668580 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.668550 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:41.760375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.760287 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca996665-95bf-48ec-982c-73306576f87e-kube-api-access-qfbtd\" (UniqueName: \"kubernetes.io/projected/ca996665-95bf-48ec-982c-73306576f87e-kube-api-access-qfbtd\") pod \"ca996665-95bf-48ec-982c-73306576f87e\" (UID: \"ca996665-95bf-48ec-982c-73306576f87e\") " Nov 02 08:39:41.760603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.760587 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca996665-95bf-48ec-982c-73306576f87e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca996665-95bf-48ec-982c-73306576f87e-plugins-config-volume\") pod \"ca996665-95bf-48ec-982c-73306576f87e\" (UID: \"ca996665-95bf-48ec-982c-73306576f87e\") " Nov 02 08:39:41.760959 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.760924 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca996665-95bf-48ec-982c-73306576f87e-plugins-config-volume" pod "ca996665-95bf-48ec-982c-73306576f87e" (UID: "ca996665-95bf-48ec-982c-73306576f87e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:41.761215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.761159 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwmlm\" (UniqueName: \"kubernetes.io/projected/f9887717-ef97-4158-95c7-dc9149c57caf-kube-api-access-zwmlm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:41.761215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.761196 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca996665-95bf-48ec-982c-73306576f87e-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:41.763232 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.763204 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca996665-95bf-48ec-982c-73306576f87e-kube-api-access-qfbtd" pod "ca996665-95bf-48ec-982c-73306576f87e" (UID: "ca996665-95bf-48ec-982c-73306576f87e"). InnerVolumeSpecName "kube-api-access-qfbtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:41.773695 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.773669 6160 generic.go:356] "Generic (PLEG): container finished" podID="ca996665-95bf-48ec-982c-73306576f87e" containerID="28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6" exitCode=0 Nov 02 08:39:41.773832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.773732 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" Nov 02 08:39:41.773832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.773750 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" event={"ID":"ca996665-95bf-48ec-982c-73306576f87e","Type":"ContainerDied","Data":"28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6"} Nov 02 08:39:41.773946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.773840 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86" event={"ID":"ca996665-95bf-48ec-982c-73306576f87e","Type":"ContainerDied","Data":"063655331f8192d9006b09c80577e29b2b58056e73f4adee73421ec9d5653ccb"} Nov 02 08:39:41.773946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.773856 6160 scope.go:122] "RemoveContainer" containerID="28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6" Nov 02 08:39:41.775831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.775394 6160 generic.go:356] "Generic (PLEG): container finished" podID="f9887717-ef97-4158-95c7-dc9149c57caf" containerID="390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271" exitCode=0 Nov 02 08:39:41.775831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.775430 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" event={"ID":"f9887717-ef97-4158-95c7-dc9149c57caf","Type":"ContainerDied","Data":"390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271"} Nov 02 08:39:41.775831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.775450 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" Nov 02 08:39:41.775831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.775480 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg" event={"ID":"f9887717-ef97-4158-95c7-dc9149c57caf","Type":"ContainerDied","Data":"051990674b6f2ea50b34ae272602cf471eff849cad112a5fc395c6041042496f"} Nov 02 08:39:41.792081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.792054 6160 scope.go:122] "RemoveContainer" containerID="28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6" Nov 02 08:39:41.792441 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:39:41.792419 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6\": container with ID starting with 28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6 not found: ID does not exist" containerID="28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6" Nov 02 08:39:41.792517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.792449 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6"} err="failed to get container status \"28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6\": rpc error: code = NotFound desc = could not find container \"28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6\": container with ID starting with 28493b475a6b30bc9b21254ad930fb05c7885632e7b36e9ef69fb96ebfbdd2b6 not found: ID does not exist" Nov 02 08:39:41.792517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.792467 6160 scope.go:122] "RemoveContainer" containerID="390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271" Nov 02 08:39:41.804832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.804803 6160 scope.go:122] "RemoveContainer" containerID="390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271" Nov 02 08:39:41.805125 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:39:41.805103 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271\": container with ID starting with 390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271 not found: ID does not exist" containerID="390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271" Nov 02 08:39:41.805273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.805132 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271"} err="failed to get container status \"390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271\": rpc error: code = NotFound desc = could not find container \"390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271\": container with ID starting with 390ba73b390332047aa925c2bc69fed080da1de2a0630114a197dadf80931271 not found: ID does not exist" Nov 02 08:39:41.863190 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.863126 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbtd\" (UniqueName: \"kubernetes.io/projected/ca996665-95bf-48ec-982c-73306576f87e-kube-api-access-qfbtd\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:41.939984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.939939 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86"] Nov 02 08:39:41.943897 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.943872 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-108cf9-67b58c9b95-djx86"] Nov 02 08:39:41.948786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:41.948755 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:42.052477 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:42.052393 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg"] Nov 02 08:39:42.057466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:42.057440 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-108cf9-7c4649d4cf-zmfxg"] Nov 02 08:39:42.579815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:42.579772 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:42.722542 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:42.722515 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:39:43.585782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:43.585751 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca996665-95bf-48ec-982c-73306576f87e" path="/var/lib/kubelet/pods/ca996665-95bf-48ec-982c-73306576f87e/volumes" Nov 02 08:39:43.586145 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:43.586131 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9887717-ef97-4158-95c7-dc9149c57caf" path="/var/lib/kubelet/pods/f9887717-ef97-4158-95c7-dc9149c57caf/volumes" Nov 02 08:39:43.640701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:43.640659 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:43.857147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:43.857050 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:44.563961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.563935 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 08:39:44.564075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.563986 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 08:39:44.654553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.654514 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 08:39:44.654673 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.654603 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 08:39:44.805575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.805544 6160 generic.go:356] "Generic (PLEG): container finished" podID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerID="7c5cf1fc5ed17591a21c210f7ec58036b7a135db544745b8da918837948fc088" exitCode=0 Nov 02 08:39:44.805713 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.805619 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" event={"ID":"ae14a9a9-59d8-45c5-bff3-d01064363e59","Type":"ContainerDied","Data":"7c5cf1fc5ed17591a21c210f7ec58036b7a135db544745b8da918837948fc088"} Nov 02 08:39:44.805713 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.805685 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" event={"ID":"ae14a9a9-59d8-45c5-bff3-d01064363e59","Type":"ContainerDied","Data":"3049eab91d1c9dcc9c9f01088247d4c0d754a6db53a66ed767c3e6828c7afbd4"} Nov 02 08:39:44.805833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.805712 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3049eab91d1c9dcc9c9f01088247d4c0d754a6db53a66ed767c3e6828c7afbd4" Nov 02 08:39:44.807784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.807760 6160 generic.go:356] "Generic (PLEG): container finished" podID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerID="a31639e9114b7ca946d277529f9c7cbcf8f57de9cf71a29d28fa04204df59ee9" exitCode=0 Nov 02 08:39:44.807880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.807837 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" event={"ID":"5bfaa8ae-8d03-4f3e-833a-39b9ba979100","Type":"ContainerDied","Data":"a31639e9114b7ca946d277529f9c7cbcf8f57de9cf71a29d28fa04204df59ee9"} Nov 02 08:39:44.814206 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.814192 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:39:44.841209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.841189 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:39:44.904413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904377 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-data\") pod \"ae14a9a9-59d8-45c5-bff3-d01064363e59\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " Nov 02 08:39:44.904824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904517 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-certs\") pod \"ae14a9a9-59d8-45c5-bff3-d01064363e59\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " Nov 02 08:39:44.904824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904579 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-socket\") pod \"ae14a9a9-59d8-45c5-bff3-d01064363e59\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " Nov 02 08:39:44.904824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904629 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae14a9a9-59d8-45c5-bff3-d01064363e59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae14a9a9-59d8-45c5-bff3-d01064363e59-istiod-ca-cert\") pod \"ae14a9a9-59d8-45c5-bff3-d01064363e59\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " Nov 02 08:39:44.904824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904687 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-data" pod "ae14a9a9-59d8-45c5-bff3-d01064363e59" (UID: "ae14a9a9-59d8-45c5-bff3-d01064363e59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.904824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904757 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-envoy\") pod \"ae14a9a9-59d8-45c5-bff3-d01064363e59\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " Nov 02 08:39:44.904824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904785 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-podinfo\") pod \"ae14a9a9-59d8-45c5-bff3-d01064363e59\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " Nov 02 08:39:44.905163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904829 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-certs" pod "ae14a9a9-59d8-45c5-bff3-d01064363e59" (UID: "ae14a9a9-59d8-45c5-bff3-d01064363e59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.905163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904847 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-socket" pod "ae14a9a9-59d8-45c5-bff3-d01064363e59" (UID: "ae14a9a9-59d8-45c5-bff3-d01064363e59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.905163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904867 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-credential-socket\") pod \"ae14a9a9-59d8-45c5-bff3-d01064363e59\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " Nov 02 08:39:44.905163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904903 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-token\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-token\") pod \"ae14a9a9-59d8-45c5-bff3-d01064363e59\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " Nov 02 08:39:44.905163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.904942 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-kube-api-access-qvsb5\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-kube-api-access-qvsb5\") pod \"ae14a9a9-59d8-45c5-bff3-d01064363e59\" (UID: \"ae14a9a9-59d8-45c5-bff3-d01064363e59\") " Nov 02 08:39:44.905163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.905112 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-credential-socket" pod "ae14a9a9-59d8-45c5-bff3-d01064363e59" (UID: "ae14a9a9-59d8-45c5-bff3-d01064363e59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.905163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.905146 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae14a9a9-59d8-45c5-bff3-d01064363e59-istiod-ca-cert" pod "ae14a9a9-59d8-45c5-bff3-d01064363e59" (UID: "ae14a9a9-59d8-45c5-bff3-d01064363e59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:44.905586 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.905559 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:44.905586 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.905579 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:44.905586 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.905588 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:44.905586 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.905599 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:44.905586 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.905608 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae14a9a9-59d8-45c5-bff3-d01064363e59-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:44.907352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.907333 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-token" pod "ae14a9a9-59d8-45c5-bff3-d01064363e59" (UID: "ae14a9a9-59d8-45c5-bff3-d01064363e59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:44.907462 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.907393 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-kube-api-access-qvsb5" pod "ae14a9a9-59d8-45c5-bff3-d01064363e59" (UID: "ae14a9a9-59d8-45c5-bff3-d01064363e59"). InnerVolumeSpecName "kube-api-access-qvsb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:44.907669 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.907652 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-podinfo" pod "ae14a9a9-59d8-45c5-bff3-d01064363e59" (UID: "ae14a9a9-59d8-45c5-bff3-d01064363e59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:44.907727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.907664 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-envoy" pod "ae14a9a9-59d8-45c5-bff3-d01064363e59" (UID: "ae14a9a9-59d8-45c5-bff3-d01064363e59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:44.926707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:44.926663 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" podUID="aef7da7f-e56f-447a-be43-eae7ac6e7da2" containerName="main" containerID="cri-o://403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4" gracePeriod=45 Nov 02 08:39:45.006266 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006235 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-data\") pod \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " Nov 02 08:39:45.006432 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006288 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-certs\") pod \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " Nov 02 08:39:45.006432 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006345 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-podinfo\") pod \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " Nov 02 08:39:45.006564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006508 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istiod-ca-cert\") pod \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " Nov 02 08:39:45.006564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006539 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-data" pod "5bfaa8ae-8d03-4f3e-833a-39b9ba979100" (UID: "5bfaa8ae-8d03-4f3e-833a-39b9ba979100"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:45.006690 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006671 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-token\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-token\") pod \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " Nov 02 08:39:45.006743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006709 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-credential-socket\") pod \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " Nov 02 08:39:45.006743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006709 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-certs" pod "5bfaa8ae-8d03-4f3e-833a-39b9ba979100" (UID: "5bfaa8ae-8d03-4f3e-833a-39b9ba979100"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006740 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-kube-api-access-cxxtk\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-kube-api-access-cxxtk\") pod \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006863 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-envoy\") pod \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006915 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-socket\") pod \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\" (UID: \"5bfaa8ae-8d03-4f3e-833a-39b9ba979100\") " Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006940 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istiod-ca-cert" pod "5bfaa8ae-8d03-4f3e-833a-39b9ba979100" (UID: "5bfaa8ae-8d03-4f3e-833a-39b9ba979100"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.006959 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-credential-socket" pod "5bfaa8ae-8d03-4f3e-833a-39b9ba979100" (UID: "5bfaa8ae-8d03-4f3e-833a-39b9ba979100"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007244 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-socket" pod "5bfaa8ae-8d03-4f3e-833a-39b9ba979100" (UID: "5bfaa8ae-8d03-4f3e-833a-39b9ba979100"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007573 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007598 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007615 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007630 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007644 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007657 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007671 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007684 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.007810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.007698 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvsb5\" (UniqueName: \"kubernetes.io/projected/ae14a9a9-59d8-45c5-bff3-d01064363e59-kube-api-access-qvsb5\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.009574 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.009544 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-podinfo" pod "5bfaa8ae-8d03-4f3e-833a-39b9ba979100" (UID: "5bfaa8ae-8d03-4f3e-833a-39b9ba979100"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:45.009888 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.009856 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-envoy" pod "5bfaa8ae-8d03-4f3e-833a-39b9ba979100" (UID: "5bfaa8ae-8d03-4f3e-833a-39b9ba979100"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:45.009993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.009940 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-token" pod "5bfaa8ae-8d03-4f3e-833a-39b9ba979100" (UID: "5bfaa8ae-8d03-4f3e-833a-39b9ba979100"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:45.009993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.009978 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-kube-api-access-cxxtk" pod "5bfaa8ae-8d03-4f3e-833a-39b9ba979100" (UID: "5bfaa8ae-8d03-4f3e-833a-39b9ba979100"). InnerVolumeSpecName "kube-api-access-cxxtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:45.110140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.110045 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxxtk\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-kube-api-access-cxxtk\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.110140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.110085 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.110140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.110100 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.110140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.110114 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfaa8ae-8d03-4f3e-833a-39b9ba979100-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.191604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.191578 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:39:45.312523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.312441 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-model-cache\") pod \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " Nov 02 08:39:45.312523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.312496 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-home\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-home\") pod \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " Nov 02 08:39:45.312523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.312517 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef7da7f-e56f-447a-be43-eae7ac6e7da2-kube-api-access-blwhm\" (UniqueName: \"kubernetes.io/projected/aef7da7f-e56f-447a-be43-eae7ac6e7da2-kube-api-access-blwhm\") pod \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " Nov 02 08:39:45.312806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.312538 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-dshm\") pod \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " Nov 02 08:39:45.312806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.312558 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tls-certs\") pod \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " Nov 02 08:39:45.312806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.312743 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tmp-dir\") pod \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\" (UID: \"aef7da7f-e56f-447a-be43-eae7ac6e7da2\") " Nov 02 08:39:45.312806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.312743 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-model-cache" pod "aef7da7f-e56f-447a-be43-eae7ac6e7da2" (UID: "aef7da7f-e56f-447a-be43-eae7ac6e7da2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:45.313022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.312853 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-home" pod "aef7da7f-e56f-447a-be43-eae7ac6e7da2" (UID: "aef7da7f-e56f-447a-be43-eae7ac6e7da2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:45.313022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.313012 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tmp-dir" pod "aef7da7f-e56f-447a-be43-eae7ac6e7da2" (UID: "aef7da7f-e56f-447a-be43-eae7ac6e7da2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:45.313319 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.313303 6160 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tmp-dir\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.313392 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.313322 6160 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-model-cache\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.313392 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.313331 6160 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-home\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.315502 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.315478 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-dshm" pod "aef7da7f-e56f-447a-be43-eae7ac6e7da2" (UID: "aef7da7f-e56f-447a-be43-eae7ac6e7da2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:45.315595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.315521 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tls-certs" pod "aef7da7f-e56f-447a-be43-eae7ac6e7da2" (UID: "aef7da7f-e56f-447a-be43-eae7ac6e7da2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:39:45.315648 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.315630 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef7da7f-e56f-447a-be43-eae7ac6e7da2-kube-api-access-blwhm" pod "aef7da7f-e56f-447a-be43-eae7ac6e7da2" (UID: "aef7da7f-e56f-447a-be43-eae7ac6e7da2"). InnerVolumeSpecName "kube-api-access-blwhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:45.385895 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.385842 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:45.414889 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.414856 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blwhm\" (UniqueName: \"kubernetes.io/projected/aef7da7f-e56f-447a-be43-eae7ac6e7da2-kube-api-access-blwhm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.414889 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.414888 6160 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7da7f-e56f-447a-be43-eae7ac6e7da2-dshm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.414889 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.414898 6160 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7da7f-e56f-447a-be43-eae7ac6e7da2-tls-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:45.571864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.571758 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:45.816509 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.816479 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" Nov 02 08:39:45.816509 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.816498 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4" event={"ID":"5bfaa8ae-8d03-4f3e-833a-39b9ba979100","Type":"ContainerDied","Data":"0bfbe862119e4416bad35332157befe1567c365aa7a9300be14b0b9e82646726"} Nov 02 08:39:45.816760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.816569 6160 scope.go:122] "RemoveContainer" containerID="a31639e9114b7ca946d277529f9c7cbcf8f57de9cf71a29d28fa04204df59ee9" Nov 02 08:39:45.818332 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.818306 6160 generic.go:356] "Generic (PLEG): container finished" podID="aef7da7f-e56f-447a-be43-eae7ac6e7da2" containerID="403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4" exitCode=0 Nov 02 08:39:45.818431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.818375 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" Nov 02 08:39:45.818482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.818434 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc" Nov 02 08:39:45.818590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.818472 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" event={"ID":"aef7da7f-e56f-447a-be43-eae7ac6e7da2","Type":"ContainerDied","Data":"403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4"} Nov 02 08:39:45.818666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.818611 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44" event={"ID":"aef7da7f-e56f-447a-be43-eae7ac6e7da2","Type":"ContainerDied","Data":"c5df461a1d703b89f3debd11b471887ae3b4c938077aee9589dde35f2d8c6ac9"} Nov 02 08:39:45.831194 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.831159 6160 scope.go:122] "RemoveContainer" containerID="403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4" Nov 02 08:39:45.841129 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.841104 6160 scope.go:122] "RemoveContainer" containerID="403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4" Nov 02 08:39:45.841452 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:39:45.841417 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4\": container with ID starting with 403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4 not found: ID does not exist" containerID="403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4" Nov 02 08:39:45.841524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:45.841461 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4"} err="failed to get container status \"403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4\": rpc error: code = NotFound desc = could not find container \"403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4\": container with ID starting with 403f303aaa2f6271e0f79a4f0587b3803b873fac4eab760132c3f7ce0c5e12f4 not found: ID does not exist" Nov 02 08:39:46.004163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:46.004078 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc"] Nov 02 08:39:46.008525 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:46.008496 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-k2wlc"] Nov 02 08:39:46.093113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:46.093033 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44"] Nov 02 08:39:46.098432 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:46.098406 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-108cf9/distinct-2-w3-108cf9-kserve-756d4dd668-9gw44"] Nov 02 08:39:46.206286 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:46.206248 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4"] Nov 02 08:39:46.209759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:46.209719 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-btqv4"] Nov 02 08:39:47.584777 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:47.584741 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" path="/var/lib/kubelet/pods/5bfaa8ae-8d03-4f3e-833a-39b9ba979100/volumes" Nov 02 08:39:47.585270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:47.585254 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" path="/var/lib/kubelet/pods/ae14a9a9-59d8-45c5-bff3-d01064363e59/volumes" Nov 02 08:39:47.585721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:47.585703 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef7da7f-e56f-447a-be43-eae7ac6e7da2" path="/var/lib/kubelet/pods/aef7da7f-e56f-447a-be43-eae7ac6e7da2/volumes" Nov 02 08:39:47.647696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:47.647637 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:48.481037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:48.481002 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:49.434122 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:49.434084 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:49.694070 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:49.693976 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:49.710622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:49.710591 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6"] Nov 02 08:39:49.710830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:49.710792 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerName="istio-proxy" containerID="cri-o://d86ae705362ed3492d4d3bade3950151c23ad2bbde62ce8424163e72cdde4b31" gracePeriod=30 Nov 02 08:39:49.722548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:49.722517 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp"] Nov 02 08:39:49.722756 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:49.722724 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerName="istio-proxy" containerID="cri-o://0c67e814deb57dffd54e0e055461ef35c60b8a0ceba40311f2e4af7b1a41fc74" gracePeriod=30 Nov 02 08:39:49.900095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:49.900054 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:50.567505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:50.567464 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:51.882892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:51.882845 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:52.117544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:52.117507 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:52.934833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:52.934800 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:53.795928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:53.795890 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:54.684092 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:54.684049 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:54.917056 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:54.917015 6160 generic.go:356] "Generic (PLEG): container finished" podID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerID="0c67e814deb57dffd54e0e055461ef35c60b8a0ceba40311f2e4af7b1a41fc74" exitCode=0 Nov 02 08:39:54.917330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:54.917303 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" event={"ID":"d6d1d5cd-6b48-41b0-8912-83a69debe723","Type":"ContainerDied","Data":"0c67e814deb57dffd54e0e055461ef35c60b8a0ceba40311f2e4af7b1a41fc74"} Nov 02 08:39:54.919285 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:54.919257 6160 generic.go:356] "Generic (PLEG): container finished" podID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerID="d86ae705362ed3492d4d3bade3950151c23ad2bbde62ce8424163e72cdde4b31" exitCode=0 Nov 02 08:39:54.919390 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:54.919300 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" event={"ID":"a9d527bf-dd46-4099-a154-789eb5d38ca1","Type":"ContainerDied","Data":"d86ae705362ed3492d4d3bade3950151c23ad2bbde62ce8424163e72cdde4b31"} Nov 02 08:39:54.996016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:54.995988 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:39:55.035429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.035402 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:39:55.064911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.064880 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d1d5cd-6b48-41b0-8912-83a69debe723-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d1d5cd-6b48-41b0-8912-83a69debe723-istiod-ca-cert\") pod \"d6d1d5cd-6b48-41b0-8912-83a69debe723\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " Nov 02 08:39:55.065074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.064937 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-kube-api-access-r59jb\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-kube-api-access-r59jb\") pod \"d6d1d5cd-6b48-41b0-8912-83a69debe723\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " Nov 02 08:39:55.065074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.064962 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-data\") pod \"a9d527bf-dd46-4099-a154-789eb5d38ca1\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " Nov 02 08:39:55.065237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065078 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-credential-socket\") pod \"a9d527bf-dd46-4099-a154-789eb5d38ca1\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " Nov 02 08:39:55.065237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065145 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-envoy\") pod \"a9d527bf-dd46-4099-a154-789eb5d38ca1\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " Nov 02 08:39:55.065237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065214 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-data" pod "a9d527bf-dd46-4099-a154-789eb5d38ca1" (UID: "a9d527bf-dd46-4099-a154-789eb5d38ca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.065383 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065265 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-envoy\") pod \"d6d1d5cd-6b48-41b0-8912-83a69debe723\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " Nov 02 08:39:55.065383 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065297 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-certs\") pod \"a9d527bf-dd46-4099-a154-789eb5d38ca1\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " Nov 02 08:39:55.065501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065298 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d1d5cd-6b48-41b0-8912-83a69debe723-istiod-ca-cert" pod "d6d1d5cd-6b48-41b0-8912-83a69debe723" (UID: "d6d1d5cd-6b48-41b0-8912-83a69debe723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:55.065501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065390 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-credential-socket" pod "a9d527bf-dd46-4099-a154-789eb5d38ca1" (UID: "a9d527bf-dd46-4099-a154-789eb5d38ca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.065501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065373 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-credential-socket\") pod \"d6d1d5cd-6b48-41b0-8912-83a69debe723\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " Nov 02 08:39:55.065662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065504 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-token\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-token\") pod \"a9d527bf-dd46-4099-a154-789eb5d38ca1\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " Nov 02 08:39:55.065662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065583 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-token\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-token\") pod \"d6d1d5cd-6b48-41b0-8912-83a69debe723\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " Nov 02 08:39:55.065662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065594 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-certs" pod "a9d527bf-dd46-4099-a154-789eb5d38ca1" (UID: "a9d527bf-dd46-4099-a154-789eb5d38ca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.065662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065603 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-credential-socket" pod "d6d1d5cd-6b48-41b0-8912-83a69debe723" (UID: "d6d1d5cd-6b48-41b0-8912-83a69debe723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.065662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065620 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-socket\") pod \"d6d1d5cd-6b48-41b0-8912-83a69debe723\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " Nov 02 08:39:55.065913 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065670 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d527bf-dd46-4099-a154-789eb5d38ca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d527bf-dd46-4099-a154-789eb5d38ca1-istiod-ca-cert\") pod \"a9d527bf-dd46-4099-a154-789eb5d38ca1\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " Nov 02 08:39:55.065913 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065791 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-podinfo\") pod \"a9d527bf-dd46-4099-a154-789eb5d38ca1\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " Nov 02 08:39:55.066025 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065979 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-socket\") pod \"a9d527bf-dd46-4099-a154-789eb5d38ca1\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " Nov 02 08:39:55.066025 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066006 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-kube-api-access-mbm4h\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-kube-api-access-mbm4h\") pod \"a9d527bf-dd46-4099-a154-789eb5d38ca1\" (UID: \"a9d527bf-dd46-4099-a154-789eb5d38ca1\") " Nov 02 08:39:55.066138 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066043 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-data\") pod \"d6d1d5cd-6b48-41b0-8912-83a69debe723\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " Nov 02 08:39:55.066138 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066125 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-certs\") pod \"d6d1d5cd-6b48-41b0-8912-83a69debe723\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " Nov 02 08:39:55.066281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066206 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-podinfo\") pod \"d6d1d5cd-6b48-41b0-8912-83a69debe723\" (UID: \"d6d1d5cd-6b48-41b0-8912-83a69debe723\") " Nov 02 08:39:55.066594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.065840 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-socket" pod "d6d1d5cd-6b48-41b0-8912-83a69debe723" (UID: "d6d1d5cd-6b48-41b0-8912-83a69debe723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.066594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066275 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d527bf-dd46-4099-a154-789eb5d38ca1-istiod-ca-cert" pod "a9d527bf-dd46-4099-a154-789eb5d38ca1" (UID: "a9d527bf-dd46-4099-a154-789eb5d38ca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:55.066812 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066785 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-data" pod "d6d1d5cd-6b48-41b0-8912-83a69debe723" (UID: "d6d1d5cd-6b48-41b0-8912-83a69debe723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.066900 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066878 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d1d5cd-6b48-41b0-8912-83a69debe723-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.066957 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066905 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.066957 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066920 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.066957 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066934 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.066957 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066950 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.067203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066965 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.067203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066978 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d527bf-dd46-4099-a154-789eb5d38ca1-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.067203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.066945 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-socket" pod "a9d527bf-dd46-4099-a154-789eb5d38ca1" (UID: "a9d527bf-dd46-4099-a154-789eb5d38ca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.067553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.067284 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-certs" pod "d6d1d5cd-6b48-41b0-8912-83a69debe723" (UID: "d6d1d5cd-6b48-41b0-8912-83a69debe723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.068469 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.068447 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-envoy" pod "a9d527bf-dd46-4099-a154-789eb5d38ca1" (UID: "a9d527bf-dd46-4099-a154-789eb5d38ca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.068577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.068484 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-token" pod "d6d1d5cd-6b48-41b0-8912-83a69debe723" (UID: "d6d1d5cd-6b48-41b0-8912-83a69debe723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:55.068577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.068497 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-envoy" pod "d6d1d5cd-6b48-41b0-8912-83a69debe723" (UID: "d6d1d5cd-6b48-41b0-8912-83a69debe723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:39:55.068577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.068518 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-kube-api-access-r59jb" pod "d6d1d5cd-6b48-41b0-8912-83a69debe723" (UID: "d6d1d5cd-6b48-41b0-8912-83a69debe723"). InnerVolumeSpecName "kube-api-access-r59jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:55.069053 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.069026 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-kube-api-access-mbm4h" pod "a9d527bf-dd46-4099-a154-789eb5d38ca1" (UID: "a9d527bf-dd46-4099-a154-789eb5d38ca1"). InnerVolumeSpecName "kube-api-access-mbm4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:55.069456 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.069422 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-podinfo" pod "d6d1d5cd-6b48-41b0-8912-83a69debe723" (UID: "d6d1d5cd-6b48-41b0-8912-83a69debe723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:55.069550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.069454 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-podinfo" pod "a9d527bf-dd46-4099-a154-789eb5d38ca1" (UID: "a9d527bf-dd46-4099-a154-789eb5d38ca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:39:55.069550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.069469 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-token" pod "a9d527bf-dd46-4099-a154-789eb5d38ca1" (UID: "a9d527bf-dd46-4099-a154-789eb5d38ca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:55.168875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168839 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.168875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168873 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.169088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168886 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.169088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168901 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.169088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168915 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbm4h\" (UniqueName: \"kubernetes.io/projected/a9d527bf-dd46-4099-a154-789eb5d38ca1-kube-api-access-mbm4h\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.169088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168929 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.169088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168941 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.169088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168953 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.169088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168968 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59jb\" (UniqueName: \"kubernetes.io/projected/d6d1d5cd-6b48-41b0-8912-83a69debe723-kube-api-access-r59jb\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.169088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168980 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d527bf-dd46-4099-a154-789eb5d38ca1-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.169088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.168992 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d1d5cd-6b48-41b0-8912-83a69debe723-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:39:55.671818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.671785 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:55.927668 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.927642 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" Nov 02 08:39:55.928170 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.927677 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp" event={"ID":"d6d1d5cd-6b48-41b0-8912-83a69debe723","Type":"ContainerDied","Data":"3f26bb90048187d3bcdd9d749dded63ac3dde2fcb5b18c09f085b809d56f4e70"} Nov 02 08:39:55.928170 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.927740 6160 scope.go:122] "RemoveContainer" containerID="0c67e814deb57dffd54e0e055461ef35c60b8a0ceba40311f2e4af7b1a41fc74" Nov 02 08:39:55.929328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.929314 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" Nov 02 08:39:55.929440 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.929329 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6" event={"ID":"a9d527bf-dd46-4099-a154-789eb5d38ca1","Type":"ContainerDied","Data":"063329750c33f04cf5e67605e72daed498ab92765cc721324b02a78e50f80fd6"} Nov 02 08:39:55.938802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:55.938783 6160 scope.go:122] "RemoveContainer" containerID="d86ae705362ed3492d4d3bade3950151c23ad2bbde62ce8424163e72cdde4b31" Nov 02 08:39:56.130195 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:56.130150 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp"] Nov 02 08:39:56.175419 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:56.175392 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-fvkzp"] Nov 02 08:39:56.237901 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:56.237807 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6"] Nov 02 08:39:56.248978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:56.248951 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b9eea3-openshift-default-557d9d8589-f2np6"] Nov 02 08:39:57.587014 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:57.586981 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" path="/var/lib/kubelet/pods/a9d527bf-dd46-4099-a154-789eb5d38ca1/volumes" Nov 02 08:39:57.587507 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:57.587491 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" path="/var/lib/kubelet/pods/d6d1d5cd-6b48-41b0-8912-83a69debe723/volumes" Nov 02 08:39:57.998772 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:39:57.998736 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:00.097878 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.097837 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:00.308894 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.308860 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:00.494822 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.494773 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:00.725973 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.725938 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm"] Nov 02 08:40:00.726155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.726134 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" containerID="cri-o://6ed9225c50c68e485845917cb0795a77b0f06f65ee2367376c9966f2ef484dcf" gracePeriod=30 Nov 02 08:40:00.738498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.738466 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4"] Nov 02 08:40:00.738725 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.738702 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" containerID="cri-o://991ac18cb46aed4599ed03c2188ee57c14a2d010554684a821b792b9b2abbe02" gracePeriod=30 Nov 02 08:40:00.756198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.756107 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258"] Nov 02 08:40:00.756412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.756356 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" containerID="cri-o://7a1082c0932fa4a2f2dd1ebd44cb44355179caf5d4f6bead0ffc10136059934c" gracePeriod=30 Nov 02 08:40:00.770896 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:00.770866 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:01.300273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:01.300240 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:01.484780 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:01.484729 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:02.639410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:02.639372 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2"] Nov 02 08:40:02.744275 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:02.744244 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6"] Nov 02 08:40:03.062602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:03.062561 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:03.285595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:03.285555 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:03.285864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:03.285842 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:40:03.285925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:03.285900 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:40:03.719031 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:03.718993 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-pz258 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:40:03.719454 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:03.719067 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:40:03.943630 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:03.943590 6160 patch_prober.go:29] interesting pod/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:40:03.943816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:03.943657 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:40:05.767028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:05.766992 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:06.010450 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.010416 6160 generic.go:356] "Generic (PLEG): container finished" podID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerID="991ac18cb46aed4599ed03c2188ee57c14a2d010554684a821b792b9b2abbe02" exitCode=0 Nov 02 08:40:06.010566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.010442 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" event={"ID":"0bba8a35-6019-4aa7-861d-2bb576831fef","Type":"ContainerDied","Data":"991ac18cb46aed4599ed03c2188ee57c14a2d010554684a821b792b9b2abbe02"} Nov 02 08:40:06.014322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.012267 6160 generic.go:356] "Generic (PLEG): container finished" podID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerID="6ed9225c50c68e485845917cb0795a77b0f06f65ee2367376c9966f2ef484dcf" exitCode=0 Nov 02 08:40:06.014322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.012353 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" event={"ID":"ff836070-4c5d-4e76-bf8b-501c47770f50","Type":"ContainerDied","Data":"6ed9225c50c68e485845917cb0795a77b0f06f65ee2367376c9966f2ef484dcf"} Nov 02 08:40:06.015167 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.015141 6160 generic.go:356] "Generic (PLEG): container finished" podID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerID="7a1082c0932fa4a2f2dd1ebd44cb44355179caf5d4f6bead0ffc10136059934c" exitCode=0 Nov 02 08:40:06.015451 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.015428 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" event={"ID":"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb","Type":"ContainerDied","Data":"7a1082c0932fa4a2f2dd1ebd44cb44355179caf5d4f6bead0ffc10136059934c"} Nov 02 08:40:06.040795 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.040763 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:40:06.063306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.063271 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:40:06.086225 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.086200 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:40:06.109593 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.109570 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-envoy\") pod \"ff836070-4c5d-4e76-bf8b-501c47770f50\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " Nov 02 08:40:06.109736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.109616 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-credential-socket\") pod \"0bba8a35-6019-4aa7-861d-2bb576831fef\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " Nov 02 08:40:06.109736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.109686 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-kube-api-access-d7fw6\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-kube-api-access-d7fw6\") pod \"ff836070-4c5d-4e76-bf8b-501c47770f50\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " Nov 02 08:40:06.109736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.109718 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-certs\") pod \"ff836070-4c5d-4e76-bf8b-501c47770f50\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " Nov 02 08:40:06.109992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.109748 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-podinfo\") pod \"ff836070-4c5d-4e76-bf8b-501c47770f50\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " Nov 02 08:40:06.109992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.109770 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-data\") pod \"0bba8a35-6019-4aa7-861d-2bb576831fef\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " Nov 02 08:40:06.109992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.109856 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-token\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-token\") pod \"0bba8a35-6019-4aa7-861d-2bb576831fef\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " Nov 02 08:40:06.109992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.109919 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-credential-socket" pod "0bba8a35-6019-4aa7-861d-2bb576831fef" (UID: "0bba8a35-6019-4aa7-861d-2bb576831fef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.110256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.109996 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-certs" pod "ff836070-4c5d-4e76-bf8b-501c47770f50" (UID: "ff836070-4c5d-4e76-bf8b-501c47770f50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.110256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110097 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-data" pod "0bba8a35-6019-4aa7-861d-2bb576831fef" (UID: "0bba8a35-6019-4aa7-861d-2bb576831fef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.110256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110232 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-certs\") pod \"0bba8a35-6019-4aa7-861d-2bb576831fef\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " Nov 02 08:40:06.110420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110339 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-token\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-token\") pod \"ff836070-4c5d-4e76-bf8b-501c47770f50\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " Nov 02 08:40:06.110420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110394 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-socket\") pod \"0bba8a35-6019-4aa7-861d-2bb576831fef\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " Nov 02 08:40:06.110531 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110496 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-certs" pod "0bba8a35-6019-4aa7-861d-2bb576831fef" (UID: "0bba8a35-6019-4aa7-861d-2bb576831fef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.110592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110493 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-envoy\") pod \"0bba8a35-6019-4aa7-861d-2bb576831fef\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " Nov 02 08:40:06.110753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110698 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-socket" pod "0bba8a35-6019-4aa7-861d-2bb576831fef" (UID: "0bba8a35-6019-4aa7-861d-2bb576831fef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.110753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110701 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-podinfo\") pod \"0bba8a35-6019-4aa7-861d-2bb576831fef\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " Nov 02 08:40:06.110886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110775 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-socket\") pod \"ff836070-4c5d-4e76-bf8b-501c47770f50\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " Nov 02 08:40:06.110886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110810 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-credential-socket\") pod \"ff836070-4c5d-4e76-bf8b-501c47770f50\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " Nov 02 08:40:06.110886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110854 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff836070-4c5d-4e76-bf8b-501c47770f50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff836070-4c5d-4e76-bf8b-501c47770f50-istiod-ca-cert\") pod \"ff836070-4c5d-4e76-bf8b-501c47770f50\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " Nov 02 08:40:06.111040 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110909 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bba8a35-6019-4aa7-861d-2bb576831fef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bba8a35-6019-4aa7-861d-2bb576831fef-istiod-ca-cert\") pod \"0bba8a35-6019-4aa7-861d-2bb576831fef\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " Nov 02 08:40:06.111040 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.110946 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-data\") pod \"ff836070-4c5d-4e76-bf8b-501c47770f50\" (UID: \"ff836070-4c5d-4e76-bf8b-501c47770f50\") " Nov 02 08:40:06.111040 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111017 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-kube-api-access-8kf2t\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-kube-api-access-8kf2t\") pod \"0bba8a35-6019-4aa7-861d-2bb576831fef\" (UID: \"0bba8a35-6019-4aa7-861d-2bb576831fef\") " Nov 02 08:40:06.111226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111046 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-socket" pod "ff836070-4c5d-4e76-bf8b-501c47770f50" (UID: "ff836070-4c5d-4e76-bf8b-501c47770f50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.111477 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111440 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-data" pod "ff836070-4c5d-4e76-bf8b-501c47770f50" (UID: "ff836070-4c5d-4e76-bf8b-501c47770f50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.111669 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111631 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.111669 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111657 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.111831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111671 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.111831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111686 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.111831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111699 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.111831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111713 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.111831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111727 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.112303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.111852 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-credential-socket" pod "ff836070-4c5d-4e76-bf8b-501c47770f50" (UID: "ff836070-4c5d-4e76-bf8b-501c47770f50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.112303 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.112011 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bba8a35-6019-4aa7-861d-2bb576831fef-istiod-ca-cert" pod "0bba8a35-6019-4aa7-861d-2bb576831fef" (UID: "0bba8a35-6019-4aa7-861d-2bb576831fef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:06.112442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.112364 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff836070-4c5d-4e76-bf8b-501c47770f50-istiod-ca-cert" pod "ff836070-4c5d-4e76-bf8b-501c47770f50" (UID: "ff836070-4c5d-4e76-bf8b-501c47770f50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:06.112544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.112517 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-envoy" pod "ff836070-4c5d-4e76-bf8b-501c47770f50" (UID: "ff836070-4c5d-4e76-bf8b-501c47770f50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.112740 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.112693 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-kube-api-access-d7fw6" pod "ff836070-4c5d-4e76-bf8b-501c47770f50" (UID: "ff836070-4c5d-4e76-bf8b-501c47770f50"). InnerVolumeSpecName "kube-api-access-d7fw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.112809 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.112787 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-token" pod "0bba8a35-6019-4aa7-861d-2bb576831fef" (UID: "0bba8a35-6019-4aa7-861d-2bb576831fef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.113088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.113066 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-podinfo" pod "ff836070-4c5d-4e76-bf8b-501c47770f50" (UID: "ff836070-4c5d-4e76-bf8b-501c47770f50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:06.113546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.113524 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-podinfo" pod "0bba8a35-6019-4aa7-861d-2bb576831fef" (UID: "0bba8a35-6019-4aa7-861d-2bb576831fef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:06.113749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.113727 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-kube-api-access-8kf2t" pod "0bba8a35-6019-4aa7-861d-2bb576831fef" (UID: "0bba8a35-6019-4aa7-861d-2bb576831fef"). InnerVolumeSpecName "kube-api-access-8kf2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.114743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.114727 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-token" pod "ff836070-4c5d-4e76-bf8b-501c47770f50" (UID: "ff836070-4c5d-4e76-bf8b-501c47770f50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.114743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.114734 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-envoy" pod "0bba8a35-6019-4aa7-861d-2bb576831fef" (UID: "0bba8a35-6019-4aa7-861d-2bb576831fef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.212580 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.212543 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istiod-ca-cert\") pod \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " Nov 02 08:40:06.212776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.212632 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-podinfo\") pod \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " Nov 02 08:40:06.212776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.212691 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-data\") pod \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " Nov 02 08:40:06.212776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.212741 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-envoy\") pod \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " Nov 02 08:40:06.212954 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.212873 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-certs\") pod \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " Nov 02 08:40:06.212954 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.212943 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-token\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-token\") pod \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " Nov 02 08:40:06.213065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.212948 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istiod-ca-cert" pod "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" (UID: "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:06.213065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213012 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-credential-socket\") pod \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " Nov 02 08:40:06.213065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213054 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-socket\") pod \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " Nov 02 08:40:06.213249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213086 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-kube-api-access-tdvdf\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-kube-api-access-tdvdf\") pod \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\" (UID: \"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb\") " Nov 02 08:40:06.213249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213230 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-data" pod "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" (UID: "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.213704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213677 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213710 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213725 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213742 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bba8a35-6019-4aa7-861d-2bb576831fef-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213757 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213769 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff836070-4c5d-4e76-bf8b-501c47770f50-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213782 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bba8a35-6019-4aa7-861d-2bb576831fef-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213796 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kf2t\" (UniqueName: \"kubernetes.io/projected/0bba8a35-6019-4aa7-861d-2bb576831fef-kube-api-access-8kf2t\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213810 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213823 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213836 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.213848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213850 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7fw6\" (UniqueName: \"kubernetes.io/projected/ff836070-4c5d-4e76-bf8b-501c47770f50-kube-api-access-d7fw6\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.214458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.213864 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff836070-4c5d-4e76-bf8b-501c47770f50-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.214788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.214752 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-certs" pod "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" (UID: "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.214788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.214765 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-socket" pod "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" (UID: "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.214948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.214927 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-credential-socket" pod "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" (UID: "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.216079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.216041 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-podinfo" pod "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" (UID: "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:06.216715 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.216688 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-kube-api-access-tdvdf" pod "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" (UID: "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb"). InnerVolumeSpecName "kube-api-access-tdvdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.217072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.217047 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-token" pod "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" (UID: "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:06.217571 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.217553 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-envoy" pod "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" (UID: "fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:06.315865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.315381 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.315865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.315415 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.315865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.315431 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.315865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.315446 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvdf\" (UniqueName: \"kubernetes.io/projected/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-kube-api-access-tdvdf\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.315865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.315461 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.315865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.315475 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:06.315865 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:06.315487 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:07.023814 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.023727 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" Nov 02 08:40:07.024259 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.023731 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258" event={"ID":"fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb","Type":"ContainerDied","Data":"7b3d1394fefa14149aa27dd931d25444612a876d32d28adcd0eab1a8c8a11224"} Nov 02 08:40:07.024259 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.023874 6160 scope.go:122] "RemoveContainer" containerID="7a1082c0932fa4a2f2dd1ebd44cb44355179caf5d4f6bead0ffc10136059934c" Nov 02 08:40:07.025652 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.025633 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" Nov 02 08:40:07.025765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.025664 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4" event={"ID":"0bba8a35-6019-4aa7-861d-2bb576831fef","Type":"ContainerDied","Data":"aa1841358e6c17856d79e6d7551b43fc7c056369238062e6d0fceb821eec2856"} Nov 02 08:40:07.027479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.027451 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" event={"ID":"ff836070-4c5d-4e76-bf8b-501c47770f50","Type":"ContainerDied","Data":"7dc3953a5ca578073d1555edd76400f9cd8cba2ac581fa05613eacb25e05e797"} Nov 02 08:40:07.027479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.027470 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm" Nov 02 08:40:07.035621 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.035606 6160 scope.go:122] "RemoveContainer" containerID="991ac18cb46aed4599ed03c2188ee57c14a2d010554684a821b792b9b2abbe02" Nov 02 08:40:07.047119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.047099 6160 scope.go:122] "RemoveContainer" containerID="6ed9225c50c68e485845917cb0795a77b0f06f65ee2367376c9966f2ef484dcf" Nov 02 08:40:07.179216 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.179184 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258"] Nov 02 08:40:07.206420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.206379 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:07.225324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.225294 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-pz258"] Nov 02 08:40:07.289681 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.289505 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4"] Nov 02 08:40:07.293530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.293502 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-24dg4"] Nov 02 08:40:07.395609 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.395572 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm"] Nov 02 08:40:07.398675 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.398655 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cba6f9af-openshift-default-db985b657-8shrm"] Nov 02 08:40:07.444462 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.444426 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:07.584630 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.584534 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" path="/var/lib/kubelet/pods/0bba8a35-6019-4aa7-861d-2bb576831fef/volumes" Nov 02 08:40:07.585390 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.585365 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" path="/var/lib/kubelet/pods/fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb/volumes" Nov 02 08:40:07.585828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.585814 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" path="/var/lib/kubelet/pods/ff836070-4c5d-4e76-bf8b-501c47770f50/volumes" Nov 02 08:40:07.639662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.639627 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" podUID="a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" containerName="payload-processing" containerID="cri-o://59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080" gracePeriod=25 Nov 02 08:40:07.745591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:07.745553 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" podUID="a252ee0e-cb81-4412-9e74-6f046758501d" containerName="payload-pre-processing" containerID="cri-o://4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff" gracePeriod=25 Nov 02 08:40:08.033739 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:08.033694 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:08.215445 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:08.215410 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:08.757120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:08.757078 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:08.984584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:08.984547 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:10.030424 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:10.030388 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:10.119622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:10.119581 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck"] Nov 02 08:40:10.119824 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:10.119803 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" containerID="cri-o://f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c" gracePeriod=30 Nov 02 08:40:10.521646 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:40:10.521606 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:40:10.566891 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:10.566855 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:10.748764 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:10.748724 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" podUID="a252ee0e-cb81-4412-9e74-6f046758501d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:40:10.750744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:10.750705 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" podUID="a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:40:11.410155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:11.410121 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn"] Nov 02 08:40:11.410611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:11.410405 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" containerID="cri-o://4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e" gracePeriod=30 Nov 02 08:40:11.425279 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:11.425245 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:11.500086 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:11.500054 6160 patch_prober.go:29] interesting pod/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:40:11.500273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:11.500107 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:40:11.857928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:11.857897 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb"] Nov 02 08:40:11.858156 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:11.858130 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" containerID="cri-o://6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5" gracePeriod=30 Nov 02 08:40:12.422261 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:12.422225 6160 patch_prober.go:29] interesting pod/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:40:12.422630 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:12.422292 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:40:12.853324 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:12.853230 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:13.052069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:13.052030 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:13.745561 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:13.745520 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:14.947593 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:14.947555 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:15.385655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.385625 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:40:15.533201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533091 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d822da84-b8e1-43c1-b841-6a220c658d8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d822da84-b8e1-43c1-b841-6a220c658d8c-istiod-ca-cert\") pod \"d822da84-b8e1-43c1-b841-6a220c658d8c\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " Nov 02 08:40:15.533201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533154 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-data\") pod \"d822da84-b8e1-43c1-b841-6a220c658d8c\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " Nov 02 08:40:15.533464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533217 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-credential-socket\") pod \"d822da84-b8e1-43c1-b841-6a220c658d8c\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " Nov 02 08:40:15.533464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533301 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-kube-api-access-prc7f\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-kube-api-access-prc7f\") pod \"d822da84-b8e1-43c1-b841-6a220c658d8c\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " Nov 02 08:40:15.533464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533365 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-podinfo\") pod \"d822da84-b8e1-43c1-b841-6a220c658d8c\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " Nov 02 08:40:15.533464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533392 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-socket\") pod \"d822da84-b8e1-43c1-b841-6a220c658d8c\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " Nov 02 08:40:15.533464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533454 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-envoy\") pod \"d822da84-b8e1-43c1-b841-6a220c658d8c\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " Nov 02 08:40:15.533722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533519 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-data" pod "d822da84-b8e1-43c1-b841-6a220c658d8c" (UID: "d822da84-b8e1-43c1-b841-6a220c658d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:15.533722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533616 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-certs\") pod \"d822da84-b8e1-43c1-b841-6a220c658d8c\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " Nov 02 08:40:15.533722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533638 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d822da84-b8e1-43c1-b841-6a220c658d8c-istiod-ca-cert" pod "d822da84-b8e1-43c1-b841-6a220c658d8c" (UID: "d822da84-b8e1-43c1-b841-6a220c658d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:15.533884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533727 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-token\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-token\") pod \"d822da84-b8e1-43c1-b841-6a220c658d8c\" (UID: \"d822da84-b8e1-43c1-b841-6a220c658d8c\") " Nov 02 08:40:15.533884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533728 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-socket" pod "d822da84-b8e1-43c1-b841-6a220c658d8c" (UID: "d822da84-b8e1-43c1-b841-6a220c658d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:15.533884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.533847 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-certs" pod "d822da84-b8e1-43c1-b841-6a220c658d8c" (UID: "d822da84-b8e1-43c1-b841-6a220c658d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:15.534052 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.534030 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-credential-socket" pod "d822da84-b8e1-43c1-b841-6a220c658d8c" (UID: "d822da84-b8e1-43c1-b841-6a220c658d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:15.534293 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.534277 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:15.534367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.534296 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:15.534367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.534307 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d822da84-b8e1-43c1-b841-6a220c658d8c-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:15.534367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.534315 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:15.534367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.534324 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:15.536237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.536214 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-podinfo" pod "d822da84-b8e1-43c1-b841-6a220c658d8c" (UID: "d822da84-b8e1-43c1-b841-6a220c658d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:15.536341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.536254 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-kube-api-access-prc7f" pod "d822da84-b8e1-43c1-b841-6a220c658d8c" (UID: "d822da84-b8e1-43c1-b841-6a220c658d8c"). InnerVolumeSpecName "kube-api-access-prc7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:15.536341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.536323 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-token" pod "d822da84-b8e1-43c1-b841-6a220c658d8c" (UID: "d822da84-b8e1-43c1-b841-6a220c658d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:15.536458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.536404 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-envoy" pod "d822da84-b8e1-43c1-b841-6a220c658d8c" (UID: "d822da84-b8e1-43c1-b841-6a220c658d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:15.635492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.635455 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:15.635634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.635508 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prc7f\" (UniqueName: \"kubernetes.io/projected/d822da84-b8e1-43c1-b841-6a220c658d8c-kube-api-access-prc7f\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:15.635634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.635519 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:15.635634 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:15.635528 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d822da84-b8e1-43c1-b841-6a220c658d8c-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:16.109400 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.109362 6160 generic.go:356] "Generic (PLEG): container finished" podID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerID="f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c" exitCode=0 Nov 02 08:40:16.109832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.109455 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" Nov 02 08:40:16.109832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.109450 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" event={"ID":"d822da84-b8e1-43c1-b841-6a220c658d8c","Type":"ContainerDied","Data":"f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c"} Nov 02 08:40:16.109832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.109585 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck" event={"ID":"d822da84-b8e1-43c1-b841-6a220c658d8c","Type":"ContainerDied","Data":"4cfc4d4f247c1a7f8f369c2396244bbd591a7f001322698f44aea5f979f50bc2"} Nov 02 08:40:16.109832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.109611 6160 scope.go:122] "RemoveContainer" containerID="f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c" Nov 02 08:40:16.120146 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.120126 6160 scope.go:122] "RemoveContainer" containerID="f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c" Nov 02 08:40:16.120431 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:40:16.120410 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c\": container with ID starting with f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c not found: ID does not exist" containerID="f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c" Nov 02 08:40:16.120497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.120442 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c"} err="failed to get container status \"f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c\": rpc error: code = NotFound desc = could not find container \"f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c\": container with ID starting with f53809bfa3b06e8ad48e8309aee8b45a5ec42e28e9c3e5e639b864a98dd70d6c not found: ID does not exist" Nov 02 08:40:16.322961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.322927 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck"] Nov 02 08:40:16.366833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.366755 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e6ca80-openshift-default-6b7b96448c-q4kck"] Nov 02 08:40:16.693382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.693352 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:40:16.849351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849316 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istiod-ca-cert\") pod \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " Nov 02 08:40:16.849351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849360 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-certs\") pod \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " Nov 02 08:40:16.849633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849393 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-socket\") pod \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " Nov 02 08:40:16.849633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849490 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-podinfo\") pod \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " Nov 02 08:40:16.849633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849544 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-token\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-token\") pod \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " Nov 02 08:40:16.849633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849597 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-kube-api-access-knrrn\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-kube-api-access-knrrn\") pod \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " Nov 02 08:40:16.849855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849644 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-envoy\") pod \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " Nov 02 08:40:16.849855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849673 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-data\") pod \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " Nov 02 08:40:16.849855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849700 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-credential-socket\") pod \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\" (UID: \"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac\") " Nov 02 08:40:16.849855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849741 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-certs" pod "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" (UID: "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:16.849855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849762 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istiod-ca-cert" pod "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" (UID: "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:16.849855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849793 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-socket" pod "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" (UID: "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:16.850158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.849994 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-data" pod "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" (UID: "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:16.850242 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.850221 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-credential-socket" pod "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" (UID: "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:16.850301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.850275 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:16.850301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.850289 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:16.850301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.850298 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:16.850455 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.850307 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:16.852267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.852240 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-token" pod "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" (UID: "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:16.852267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.852250 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-kube-api-access-knrrn" pod "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" (UID: "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac"). InnerVolumeSpecName "kube-api-access-knrrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:16.852505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.852489 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-envoy" pod "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" (UID: "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:16.852547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.852526 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-podinfo" pod "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" (UID: "1e44d29d-9365-478a-ba4b-33b5e1b7f9ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:16.951972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.951874 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:16.951972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.951910 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:16.951972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.951923 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knrrn\" (UniqueName: \"kubernetes.io/projected/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-kube-api-access-knrrn\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:16.951972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.951938 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:16.951972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:16.951951 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.102751 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.102724 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:40:17.117697 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.117669 6160 generic.go:356] "Generic (PLEG): container finished" podID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerID="4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e" exitCode=0 Nov 02 08:40:17.118055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.117722 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" Nov 02 08:40:17.118055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.117751 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" event={"ID":"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac","Type":"ContainerDied","Data":"4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e"} Nov 02 08:40:17.118055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.117807 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn" event={"ID":"1e44d29d-9365-478a-ba4b-33b5e1b7f9ac","Type":"ContainerDied","Data":"c2044c5b033da28bd681470df59ccd7455f0e95e91fcc5e37df43b468bfd2894"} Nov 02 08:40:17.118055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.117828 6160 scope.go:122] "RemoveContainer" containerID="4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e" Nov 02 08:40:17.120614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.120588 6160 generic.go:356] "Generic (PLEG): container finished" podID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerID="6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5" exitCode=0 Nov 02 08:40:17.120729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.120627 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" event={"ID":"04fe9b9f-aeae-452a-bf98-31440dba687c","Type":"ContainerDied","Data":"6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5"} Nov 02 08:40:17.120729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.120642 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" Nov 02 08:40:17.120729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.120665 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb" event={"ID":"04fe9b9f-aeae-452a-bf98-31440dba687c","Type":"ContainerDied","Data":"13ea2c6de1c938cd1d043322129d3f288c97d385ae69f4eaf42991f86a1581a1"} Nov 02 08:40:17.128482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.128461 6160 scope.go:122] "RemoveContainer" containerID="4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e" Nov 02 08:40:17.128769 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:40:17.128752 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e\": container with ID starting with 4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e not found: ID does not exist" containerID="4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e" Nov 02 08:40:17.128818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.128780 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e"} err="failed to get container status \"4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e\": rpc error: code = NotFound desc = could not find container \"4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e\": container with ID starting with 4924a87abe9370242a0a0c53ad7a27ad999907904c056cbcdd113bcbb742829e not found: ID does not exist" Nov 02 08:40:17.128818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.128797 6160 scope.go:122] "RemoveContainer" containerID="6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5" Nov 02 08:40:17.138268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.138252 6160 scope.go:122] "RemoveContainer" containerID="6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5" Nov 02 08:40:17.138531 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:40:17.138512 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5\": container with ID starting with 6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5 not found: ID does not exist" containerID="6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5" Nov 02 08:40:17.138566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.138539 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5"} err="failed to get container status \"6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5\": rpc error: code = NotFound desc = could not find container \"6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5\": container with ID starting with 6bfa7196c04bc49341e70f89125d8146cffcad67b5ebe6fe5a9aed44926b3cd5 not found: ID does not exist" Nov 02 08:40:17.255377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255298 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-token\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-token\") pod \"04fe9b9f-aeae-452a-bf98-31440dba687c\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " Nov 02 08:40:17.255377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255358 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-certs\") pod \"04fe9b9f-aeae-452a-bf98-31440dba687c\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " Nov 02 08:40:17.255590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255396 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-envoy\") pod \"04fe9b9f-aeae-452a-bf98-31440dba687c\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " Nov 02 08:40:17.255590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255422 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-socket\") pod \"04fe9b9f-aeae-452a-bf98-31440dba687c\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " Nov 02 08:40:17.255590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255464 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-data\") pod \"04fe9b9f-aeae-452a-bf98-31440dba687c\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " Nov 02 08:40:17.255590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255505 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-credential-socket\") pod \"04fe9b9f-aeae-452a-bf98-31440dba687c\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " Nov 02 08:40:17.255590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255544 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-podinfo\") pod \"04fe9b9f-aeae-452a-bf98-31440dba687c\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " Nov 02 08:40:17.255818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255649 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fe9b9f-aeae-452a-bf98-31440dba687c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe9b9f-aeae-452a-bf98-31440dba687c-istiod-ca-cert\") pod \"04fe9b9f-aeae-452a-bf98-31440dba687c\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " Nov 02 08:40:17.255818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255692 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-kube-api-access-tb6hf\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-kube-api-access-tb6hf\") pod \"04fe9b9f-aeae-452a-bf98-31440dba687c\" (UID: \"04fe9b9f-aeae-452a-bf98-31440dba687c\") " Nov 02 08:40:17.255922 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255862 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-certs" pod "04fe9b9f-aeae-452a-bf98-31440dba687c" (UID: "04fe9b9f-aeae-452a-bf98-31440dba687c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:17.255922 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255880 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-data" pod "04fe9b9f-aeae-452a-bf98-31440dba687c" (UID: "04fe9b9f-aeae-452a-bf98-31440dba687c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:17.256021 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.255961 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-socket" pod "04fe9b9f-aeae-452a-bf98-31440dba687c" (UID: "04fe9b9f-aeae-452a-bf98-31440dba687c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:17.256195 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.256146 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-credential-socket" pod "04fe9b9f-aeae-452a-bf98-31440dba687c" (UID: "04fe9b9f-aeae-452a-bf98-31440dba687c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:17.256326 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.256242 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fe9b9f-aeae-452a-bf98-31440dba687c-istiod-ca-cert" pod "04fe9b9f-aeae-452a-bf98-31440dba687c" (UID: "04fe9b9f-aeae-452a-bf98-31440dba687c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:17.256404 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.256382 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe9b9f-aeae-452a-bf98-31440dba687c-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.256466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.256402 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.256466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.256415 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.256466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.256426 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.256466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.256440 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.258197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.258155 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-token" pod "04fe9b9f-aeae-452a-bf98-31440dba687c" (UID: "04fe9b9f-aeae-452a-bf98-31440dba687c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:17.258432 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.258409 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-envoy" pod "04fe9b9f-aeae-452a-bf98-31440dba687c" (UID: "04fe9b9f-aeae-452a-bf98-31440dba687c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:40:17.258550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.258528 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-podinfo" pod "04fe9b9f-aeae-452a-bf98-31440dba687c" (UID: "04fe9b9f-aeae-452a-bf98-31440dba687c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:40:17.258602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.258592 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-kube-api-access-tb6hf" pod "04fe9b9f-aeae-452a-bf98-31440dba687c" (UID: "04fe9b9f-aeae-452a-bf98-31440dba687c"). InnerVolumeSpecName "kube-api-access-tb6hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:17.323308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.323270 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn"] Nov 02 08:40:17.330581 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.330545 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e6407-openshift-default-5fcc88db7c-rltfn"] Nov 02 08:40:17.357496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.357464 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.357496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.357489 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.357496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.357500 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6hf\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-kube-api-access-tb6hf\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.357778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.357519 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fe9b9f-aeae-452a-bf98-31440dba687c-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:17.552031 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.551940 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb"] Nov 02 08:40:17.555533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.555503 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-10d2fb-openshift-default-7689cfdd95-kggwb"] Nov 02 08:40:17.586480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.586447 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" path="/var/lib/kubelet/pods/04fe9b9f-aeae-452a-bf98-31440dba687c/volumes" Nov 02 08:40:17.586932 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.586915 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" path="/var/lib/kubelet/pods/1e44d29d-9365-478a-ba4b-33b5e1b7f9ac/volumes" Nov 02 08:40:17.587352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:17.587338 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" path="/var/lib/kubelet/pods/d822da84-b8e1-43c1-b841-6a220c658d8c/volumes" Nov 02 08:40:19.080826 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:19.080790 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:20.748635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:20.748590 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" podUID="a252ee0e-cb81-4412-9e74-6f046758501d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:40:20.749053 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:20.748710 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" podUID="a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:40:23.878570 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:23.878529 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:24.053624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:24.053588 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:27.903910 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:27.903830 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:40:27.969018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:27.968990 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-kube-api-access-qcqzn\" (UniqueName: \"kubernetes.io/projected/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-kube-api-access-qcqzn\") pod \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\" (UID: \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\") " Nov 02 08:40:27.969213 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:27.969027 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-plugins-config-volume\") pod \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\" (UID: \"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b\") " Nov 02 08:40:27.969493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:27.969467 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-plugins-config-volume" pod "a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" (UID: "a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:27.972281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:27.972254 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-kube-api-access-qcqzn" pod "a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" (UID: "a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b"). InnerVolumeSpecName "kube-api-access-qcqzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:28.002089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.002065 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:40:28.070372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.070341 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a252ee0e-cb81-4412-9e74-6f046758501d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a252ee0e-cb81-4412-9e74-6f046758501d-plugins-config-volume\") pod \"a252ee0e-cb81-4412-9e74-6f046758501d\" (UID: \"a252ee0e-cb81-4412-9e74-6f046758501d\") " Nov 02 08:40:28.070570 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.070402 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a252ee0e-cb81-4412-9e74-6f046758501d-kube-api-access-cgdch\" (UniqueName: \"kubernetes.io/projected/a252ee0e-cb81-4412-9e74-6f046758501d-kube-api-access-cgdch\") pod \"a252ee0e-cb81-4412-9e74-6f046758501d\" (UID: \"a252ee0e-cb81-4412-9e74-6f046758501d\") " Nov 02 08:40:28.070726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.070712 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcqzn\" (UniqueName: \"kubernetes.io/projected/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-kube-api-access-qcqzn\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:28.070788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.070728 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:28.070788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.070753 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a252ee0e-cb81-4412-9e74-6f046758501d-plugins-config-volume" pod "a252ee0e-cb81-4412-9e74-6f046758501d" (UID: "a252ee0e-cb81-4412-9e74-6f046758501d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:28.072652 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.072631 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a252ee0e-cb81-4412-9e74-6f046758501d-kube-api-access-cgdch" pod "a252ee0e-cb81-4412-9e74-6f046758501d" (UID: "a252ee0e-cb81-4412-9e74-6f046758501d"). InnerVolumeSpecName "kube-api-access-cgdch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:28.171990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.171951 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a252ee0e-cb81-4412-9e74-6f046758501d-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:28.171990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.171984 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgdch\" (UniqueName: \"kubernetes.io/projected/a252ee0e-cb81-4412-9e74-6f046758501d-kube-api-access-cgdch\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:28.210244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.210209 6160 generic.go:356] "Generic (PLEG): container finished" podID="a252ee0e-cb81-4412-9e74-6f046758501d" containerID="4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff" exitCode=0 Nov 02 08:40:28.210443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.210255 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" Nov 02 08:40:28.210443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.210294 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" event={"ID":"a252ee0e-cb81-4412-9e74-6f046758501d","Type":"ContainerDied","Data":"4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff"} Nov 02 08:40:28.210443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.210355 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6" event={"ID":"a252ee0e-cb81-4412-9e74-6f046758501d","Type":"ContainerDied","Data":"a2494ccc400c362971843674d40a1bb68dcbf6cf5de441df2deedca8524aba6b"} Nov 02 08:40:28.210443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.210384 6160 scope.go:122] "RemoveContainer" containerID="4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff" Nov 02 08:40:28.212055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.212028 6160 generic.go:356] "Generic (PLEG): container finished" podID="a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" containerID="59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080" exitCode=0 Nov 02 08:40:28.212197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.212070 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" event={"ID":"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b","Type":"ContainerDied","Data":"59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080"} Nov 02 08:40:28.212197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.212112 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" event={"ID":"a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b","Type":"ContainerDied","Data":"8ff000c7f32cd6da4a7ee54a72eca96c84871cf73794ec849d7d854260a1900c"} Nov 02 08:40:28.212197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.212074 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2" Nov 02 08:40:28.223365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.223274 6160 scope.go:122] "RemoveContainer" containerID="4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff" Nov 02 08:40:28.223759 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:40:28.223733 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff\": container with ID starting with 4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff not found: ID does not exist" containerID="4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff" Nov 02 08:40:28.223864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.223765 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff"} err="failed to get container status \"4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff\": rpc error: code = NotFound desc = could not find container \"4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff\": container with ID starting with 4d071bc850e932e297193790b664fc0dd376aa394245287712f97e5321d337ff not found: ID does not exist" Nov 02 08:40:28.223864 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.223785 6160 scope.go:122] "RemoveContainer" containerID="59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080" Nov 02 08:40:28.233509 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.233486 6160 scope.go:122] "RemoveContainer" containerID="59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080" Nov 02 08:40:28.233797 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:40:28.233775 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080\": container with ID starting with 59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080 not found: ID does not exist" containerID="59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080" Nov 02 08:40:28.233842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.233806 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080"} err="failed to get container status \"59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080\": rpc error: code = NotFound desc = could not find container \"59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080\": container with ID starting with 59898c7d1f8f99c4410a90c51b8deeeba2bd5b43c16a7995490b5469c9299080 not found: ID does not exist" Nov 02 08:40:28.364232 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.364202 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6"] Nov 02 08:40:28.367935 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.367903 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9d77b6-6b8d464f8c-chxn6"] Nov 02 08:40:28.471385 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.471259 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2"] Nov 02 08:40:28.475549 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.475520 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9d77b6-869d7d4b59-9gcx2"] Nov 02 08:40:28.959105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:28.959073 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:29.589228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:29.587191 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a252ee0e-cb81-4412-9e74-6f046758501d" path="/var/lib/kubelet/pods/a252ee0e-cb81-4412-9e74-6f046758501d/volumes" Nov 02 08:40:29.589228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:29.587865 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" path="/var/lib/kubelet/pods/a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b/volumes" Nov 02 08:40:30.724118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:30.724084 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:30.934658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:30.934621 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:32.510514 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:32.510477 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:32.813796 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:32.813707 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:34.273078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:34.273041 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:34.490999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:34.490965 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:36.928198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:36.928134 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:39.166026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.165987 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5"] Nov 02 08:40:39.166538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166523 6160 cpu_manager.go:414] "containerMap: removing container" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" Nov 02 08:40:39.166585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166538 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" Nov 02 08:40:39.166585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166554 6160 cpu_manager.go:414] "containerMap: removing container" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" Nov 02 08:40:39.166585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166559 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" Nov 02 08:40:39.166585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166570 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" containerName="maas-api" Nov 02 08:40:39.166585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166575 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" containerName="maas-api" Nov 02 08:40:39.166585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166582 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" containerName="payload-processing" Nov 02 08:40:39.166585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166586 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" containerName="payload-processing" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166593 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166598 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166605 6160 cpu_manager.go:414] "containerMap: removing container" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166610 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166616 6160 cpu_manager.go:414] "containerMap: removing container" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166620 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166626 6160 cpu_manager.go:414] "containerMap: removing container" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166630 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166638 6160 cpu_manager.go:414] "containerMap: removing container" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166642 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166654 6160 cpu_manager.go:414] "containerMap: removing container" podUID="aef7da7f-e56f-447a-be43-eae7ac6e7da2" containerName="main" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166659 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef7da7f-e56f-447a-be43-eae7ac6e7da2" containerName="main" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166667 6160 cpu_manager.go:414] "containerMap: removing container" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166671 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166683 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a252ee0e-cb81-4412-9e74-6f046758501d" containerName="payload-pre-processing" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166687 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a252ee0e-cb81-4412-9e74-6f046758501d" containerName="payload-pre-processing" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166693 6160 cpu_manager.go:414] "containerMap: removing container" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166698 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166705 6160 cpu_manager.go:414] "containerMap: removing container" podUID="ca996665-95bf-48ec-982c-73306576f87e" containerName="payload-pre-processing" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166710 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca996665-95bf-48ec-982c-73306576f87e" containerName="payload-pre-processing" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166720 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166725 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166732 6160 cpu_manager.go:414] "containerMap: removing container" podUID="f9887717-ef97-4158-95c7-dc9149c57caf" containerName="payload-processing" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166737 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9887717-ef97-4158-95c7-dc9149c57caf" containerName="payload-processing" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166806 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bb8bdb-5d51-41f7-a7b9-4795d8d8ddfc" containerName="maas-api" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166815 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57eaa6d-c5c0-4126-bd0c-1c6f59c86c8b" containerName="payload-processing" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166821 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d527bf-dd46-4099-a154-789eb5d38ca1" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166827 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff836070-4c5d-4e76-bf8b-501c47770f50" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166831 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="d822da84-b8e1-43c1-b841-6a220c658d8c" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166836 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bba8a35-6019-4aa7-861d-2bb576831fef" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166842 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae14a9a9-59d8-45c5-bff3-d01064363e59" containerName="istio-proxy" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166850 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef7da7f-e56f-447a-be43-eae7ac6e7da2" containerName="main" Nov 02 08:40:39.166842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166854 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d1d5cd-6b48-41b0-8912-83a69debe723" containerName="istio-proxy" Nov 02 08:40:39.167835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166861 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="a252ee0e-cb81-4412-9e74-6f046758501d" containerName="payload-pre-processing" Nov 02 08:40:39.167835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166866 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fe9b9f-aeae-452a-bf98-31440dba687c" containerName="istio-proxy" Nov 02 08:40:39.167835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166872 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca996665-95bf-48ec-982c-73306576f87e" containerName="payload-pre-processing" Nov 02 08:40:39.167835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166877 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfaa8ae-8d03-4f3e-833a-39b9ba979100" containerName="istio-proxy" Nov 02 08:40:39.167835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166883 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9887717-ef97-4158-95c7-dc9149c57caf" containerName="payload-processing" Nov 02 08:40:39.167835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166889 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdce3e7-463e-4150-8e0f-ab3d35dfbfdb" containerName="istio-proxy" Nov 02 08:40:39.167835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.166895 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e44d29d-9365-478a-ba4b-33b5e1b7f9ac" containerName="istio-proxy" Nov 02 08:40:39.170291 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.170274 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.221815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.221786 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9ceb40-serving-cert\"" Nov 02 08:40:39.271050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.271016 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5"] Nov 02 08:40:39.300886 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.300854 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2324-5246-4559-81d8-507723e7ab6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.301063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.300894 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7nhw\" (UniqueName: \"kubernetes.io/projected/9b8d2324-5246-4559-81d8-507723e7ab6b-kube-api-access-g7nhw\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.301063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.300925 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.301063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.301053 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.402410 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.402368 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2324-5246-4559-81d8-507723e7ab6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.402633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.402422 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7nhw\" (UniqueName: \"kubernetes.io/projected/9b8d2324-5246-4559-81d8-507723e7ab6b-kube-api-access-g7nhw\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.402633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.402447 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.402633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.402530 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.402806 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:40:39.402655 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-serving-cert: secret "maas-api-e2e-ipp-a-9ceb40-serving-cert" not found Nov 02 08:40:39.402806 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:40:39.402727 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls podName:9b8d2324-5246-4559-81d8-507723e7ab6b nodeName:}" failed. No retries permitted until 2026-11-02 08:40:39.902705657 +0000 UTC m=+2038.456209832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls") pod "maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" (UID: "9b8d2324-5246-4559-81d8-507723e7ab6b") : secret "maas-api-e2e-ipp-a-9ceb40-serving-cert" not found Nov 02 08:40:39.403134 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.403109 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2324-5246-4559-81d8-507723e7ab6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.405015 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.404998 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.429026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.428935 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:39.456857 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.456820 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7nhw\" (UniqueName: \"kubernetes.io/projected/9b8d2324-5246-4559-81d8-507723e7ab6b-kube-api-access-g7nhw\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.618040 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.618001 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:39.877830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.877735 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:39.909191 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.909150 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:39.911801 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:39.911777 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:40.083612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:40.083583 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:40.464694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:40.464665 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5"] Nov 02 08:40:40.466156 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:40:40.466127 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8d2324_5246_4559_81d8_507723e7ab6b.slice/crio-a03bb644056fc0470bba7c3dd8f94b6270e2a8f0b2fa3464da1dc20e1694a818 WatchSource:0}: Error finding container a03bb644056fc0470bba7c3dd8f94b6270e2a8f0b2fa3464da1dc20e1694a818: Status 404 returned error can't find the container with id a03bb644056fc0470bba7c3dd8f94b6270e2a8f0b2fa3464da1dc20e1694a818 Nov 02 08:40:40.467676 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:40.467652 6160 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:40:40.522984 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:40:40.522951 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:40:40.622845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:40.622801 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:41.298383 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:41.298330 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:41.311685 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:41.311653 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" event={"ID":"9b8d2324-5246-4559-81d8-507723e7ab6b","Type":"ContainerStarted","Data":"a03bb644056fc0470bba7c3dd8f94b6270e2a8f0b2fa3464da1dc20e1694a818"} Nov 02 08:40:42.257262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:42.257228 6160 scope.go:122] "RemoveContainer" containerID="7c5cf1fc5ed17591a21c210f7ec58036b7a135db544745b8da918837948fc088" Nov 02 08:40:42.267582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:42.267561 6160 scope.go:122] "RemoveContainer" containerID="c6ef9174351e55a15dc58d7c2fc59ca1be2e5bb82d226c457366dd978995ee2d" Nov 02 08:40:42.326398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:42.326365 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" event={"ID":"9b8d2324-5246-4559-81d8-507723e7ab6b","Type":"ContainerStarted","Data":"7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444"} Nov 02 08:40:42.326573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:42.326551 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:42.436895 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:42.436827 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" podStartSLOduration=2.530981439 podStartE2EDuration="3.436806044s" totalImagesPullingTime="905.824605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:42.435138609 +0000 UTC m=+2040.988642826" watchObservedRunningTime="2026-11-02 08:40:42.436806044 +0000 UTC m=+2040.990310245" Nov 02 08:40:42.628042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:42.627955 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:43.257244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:43.257199 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:43.577747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:43.577641 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:44.943415 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:44.943375 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:47.822463 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:47.822427 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:48.020358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:48.020321 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:48.339947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:48.339918 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:40:48.819511 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:48.819477 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:49.639945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:49.639913 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:50.118084 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.118049 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh"] Nov 02 08:40:50.118332 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.118310 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" podUID="7ab5677e-be56-4ada-a457-7ee7d9ad4cda" containerName="payload-processing" containerID="cri-o://34a55dd861414af2675a2173746962ced44b6df7b71c8ddfc8bfdf5988c28323" gracePeriod=30 Nov 02 08:40:50.219438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.219398 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx"] Nov 02 08:40:50.219718 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.219683 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" podUID="c9a8db27-6990-4903-b0fe-73f712e5a59a" containerName="payload-pre-processing" containerID="cri-o://83c1f7c6c78a2be7152acdab95246fd67b8dde8110f011062ca0843da5444e18" gracePeriod=30 Nov 02 08:40:50.321041 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.315009 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:50.398543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.398450 6160 generic.go:356] "Generic (PLEG): container finished" podID="c9a8db27-6990-4903-b0fe-73f712e5a59a" containerID="83c1f7c6c78a2be7152acdab95246fd67b8dde8110f011062ca0843da5444e18" exitCode=0 Nov 02 08:40:50.398543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.398510 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" event={"ID":"c9a8db27-6990-4903-b0fe-73f712e5a59a","Type":"ContainerDied","Data":"83c1f7c6c78a2be7152acdab95246fd67b8dde8110f011062ca0843da5444e18"} Nov 02 08:40:50.401055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.401018 6160 generic.go:356] "Generic (PLEG): container finished" podID="7ab5677e-be56-4ada-a457-7ee7d9ad4cda" containerID="34a55dd861414af2675a2173746962ced44b6df7b71c8ddfc8bfdf5988c28323" exitCode=0 Nov 02 08:40:50.401159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.401052 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" event={"ID":"7ab5677e-be56-4ada-a457-7ee7d9ad4cda","Type":"ContainerDied","Data":"34a55dd861414af2675a2173746962ced44b6df7b71c8ddfc8bfdf5988c28323"} Nov 02 08:40:50.430617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.430591 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:40:50.480965 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.480914 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:50.503136 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.503111 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:40:50.536389 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.536364 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-plugins-config-volume\") pod \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\" (UID: \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\") " Nov 02 08:40:50.536519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.536401 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-kube-api-access-x9wmk\" (UniqueName: \"kubernetes.io/projected/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-kube-api-access-x9wmk\") pod \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\" (UID: \"7ab5677e-be56-4ada-a457-7ee7d9ad4cda\") " Nov 02 08:40:50.536725 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.536705 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-plugins-config-volume" pod "7ab5677e-be56-4ada-a457-7ee7d9ad4cda" (UID: "7ab5677e-be56-4ada-a457-7ee7d9ad4cda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:50.538691 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.538672 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-kube-api-access-x9wmk" pod "7ab5677e-be56-4ada-a457-7ee7d9ad4cda" (UID: "7ab5677e-be56-4ada-a457-7ee7d9ad4cda"). InnerVolumeSpecName "kube-api-access-x9wmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:50.637125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.637091 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a8db27-6990-4903-b0fe-73f712e5a59a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a8db27-6990-4903-b0fe-73f712e5a59a-plugins-config-volume\") pod \"c9a8db27-6990-4903-b0fe-73f712e5a59a\" (UID: \"c9a8db27-6990-4903-b0fe-73f712e5a59a\") " Nov 02 08:40:50.637366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.637346 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a8db27-6990-4903-b0fe-73f712e5a59a-kube-api-access-dcmsj\" (UniqueName: \"kubernetes.io/projected/c9a8db27-6990-4903-b0fe-73f712e5a59a-kube-api-access-dcmsj\") pod \"c9a8db27-6990-4903-b0fe-73f712e5a59a\" (UID: \"c9a8db27-6990-4903-b0fe-73f712e5a59a\") " Nov 02 08:40:50.637591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.637567 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a8db27-6990-4903-b0fe-73f712e5a59a-plugins-config-volume" pod "c9a8db27-6990-4903-b0fe-73f712e5a59a" (UID: "c9a8db27-6990-4903-b0fe-73f712e5a59a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:50.637854 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.637834 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a8db27-6990-4903-b0fe-73f712e5a59a-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:50.637930 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.637861 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:50.637930 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.637875 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9wmk\" (UniqueName: \"kubernetes.io/projected/7ab5677e-be56-4ada-a457-7ee7d9ad4cda-kube-api-access-x9wmk\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:50.640356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.640323 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a8db27-6990-4903-b0fe-73f712e5a59a-kube-api-access-dcmsj" pod "c9a8db27-6990-4903-b0fe-73f712e5a59a" (UID: "c9a8db27-6990-4903-b0fe-73f712e5a59a"). InnerVolumeSpecName "kube-api-access-dcmsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:50.738590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.738561 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmsj\" (UniqueName: \"kubernetes.io/projected/c9a8db27-6990-4903-b0fe-73f712e5a59a-kube-api-access-dcmsj\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:40:50.996896 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:50.996798 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:51.409252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.409159 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" Nov 02 08:40:51.409413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.409158 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx" event={"ID":"c9a8db27-6990-4903-b0fe-73f712e5a59a","Type":"ContainerDied","Data":"8514e6e0e6f974c868355f4cebda4ed19b152994f16653757ee047653ad0a39d"} Nov 02 08:40:51.409413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.409336 6160 scope.go:122] "RemoveContainer" containerID="83c1f7c6c78a2be7152acdab95246fd67b8dde8110f011062ca0843da5444e18" Nov 02 08:40:51.410892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.410739 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" Nov 02 08:40:51.410892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.410731 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh" event={"ID":"7ab5677e-be56-4ada-a457-7ee7d9ad4cda","Type":"ContainerDied","Data":"6a2f78a315d6a647484438353d80cb6fc5aefed4dcd7a7bd8d35196e3a7d493d"} Nov 02 08:40:51.421237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.421211 6160 scope.go:122] "RemoveContainer" containerID="34a55dd861414af2675a2173746962ced44b6df7b71c8ddfc8bfdf5988c28323" Nov 02 08:40:51.540289 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.540251 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv"] Nov 02 08:40:51.541343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.541088 6160 cpu_manager.go:414] "containerMap: removing container" podUID="7ab5677e-be56-4ada-a457-7ee7d9ad4cda" containerName="payload-processing" Nov 02 08:40:51.541343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.541112 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab5677e-be56-4ada-a457-7ee7d9ad4cda" containerName="payload-processing" Nov 02 08:40:51.541343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.541143 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c9a8db27-6990-4903-b0fe-73f712e5a59a" containerName="payload-pre-processing" Nov 02 08:40:51.541343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.541152 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a8db27-6990-4903-b0fe-73f712e5a59a" containerName="payload-pre-processing" Nov 02 08:40:51.541343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.541315 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab5677e-be56-4ada-a457-7ee7d9ad4cda" containerName="payload-processing" Nov 02 08:40:51.541343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.541331 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a8db27-6990-4903-b0fe-73f712e5a59a" containerName="payload-pre-processing" Nov 02 08:40:51.548531 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.548504 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:40:51.569823 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.569761 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx"] Nov 02 08:40:51.573773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.573715 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a9bb-dcd64c4d7-n5lsx"] Nov 02 08:40:51.585894 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.585863 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a8db27-6990-4903-b0fe-73f712e5a59a" path="/var/lib/kubelet/pods/c9a8db27-6990-4903-b0fe-73f712e5a59a/volumes" Nov 02 08:40:51.599677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.599646 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ad4580\"" Nov 02 08:40:51.599950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.599934 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ad4580-dockercfg-9h8w7\"" Nov 02 08:40:51.603691 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.603668 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv"] Nov 02 08:40:51.649847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.649806 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv\" (UID: \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:40:51.650271 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.649859 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtb5\" (UniqueName: \"kubernetes.io/projected/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-kube-api-access-mgtb5\") pod \"payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv\" (UID: \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:40:51.680427 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.679206 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh"] Nov 02 08:40:51.684759 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.684725 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a9bb-749666bffb-j8tgh"] Nov 02 08:40:51.715766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.715735 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd"] Nov 02 08:40:51.720090 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.720066 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:40:51.751415 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.751378 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv\" (UID: \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:40:51.751601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.751426 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtb5\" (UniqueName: \"kubernetes.io/projected/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-kube-api-access-mgtb5\") pod \"payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv\" (UID: \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:40:51.752062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.752039 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv\" (UID: \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:40:51.774330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.774291 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd"] Nov 02 08:40:51.805777 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.805742 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtb5\" (UniqueName: \"kubernetes.io/projected/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-kube-api-access-mgtb5\") pod \"payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv\" (UID: \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:40:51.852924 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.852892 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b737b-8aeb-47e7-920a-33a1d396b983-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd\" (UID: \"8a6b737b-8aeb-47e7-920a-33a1d396b983\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:40:51.853080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.853027 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsmqd\" (UniqueName: \"kubernetes.io/projected/8a6b737b-8aeb-47e7-920a-33a1d396b983-kube-api-access-vsmqd\") pod \"payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd\" (UID: \"8a6b737b-8aeb-47e7-920a-33a1d396b983\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:40:51.870499 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.870466 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:40:51.953952 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.953910 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b737b-8aeb-47e7-920a-33a1d396b983-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd\" (UID: \"8a6b737b-8aeb-47e7-920a-33a1d396b983\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:40:51.954132 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.953996 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsmqd\" (UniqueName: \"kubernetes.io/projected/8a6b737b-8aeb-47e7-920a-33a1d396b983-kube-api-access-vsmqd\") pod \"payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd\" (UID: \"8a6b737b-8aeb-47e7-920a-33a1d396b983\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:40:51.954939 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:51.954914 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b737b-8aeb-47e7-920a-33a1d396b983-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd\" (UID: \"8a6b737b-8aeb-47e7-920a-33a1d396b983\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:40:52.008395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.008362 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsmqd\" (UniqueName: \"kubernetes.io/projected/8a6b737b-8aeb-47e7-920a-33a1d396b983-kube-api-access-vsmqd\") pod \"payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd\" (UID: \"8a6b737b-8aeb-47e7-920a-33a1d396b983\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:40:52.033554 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.033515 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:40:52.118684 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.118652 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:52.251694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.249924 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv"] Nov 02 08:40:52.415063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.415035 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd"] Nov 02 08:40:52.417316 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:40:52.417282 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6b737b_8aeb_47e7_920a_33a1d396b983.slice/crio-123e0de7103f2a6df6350ca431ee913188d25b66e7bd98bd2fe51a7c48865f78 WatchSource:0}: Error finding container 123e0de7103f2a6df6350ca431ee913188d25b66e7bd98bd2fe51a7c48865f78: Status 404 returned error can't find the container with id 123e0de7103f2a6df6350ca431ee913188d25b66e7bd98bd2fe51a7c48865f78 Nov 02 08:40:52.420356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.420325 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" event={"ID":"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab","Type":"ContainerStarted","Data":"b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948"} Nov 02 08:40:52.420496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.420390 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" event={"ID":"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab","Type":"ContainerStarted","Data":"5109838f2950f753f323fbad09cdedf38d748c12c84251c92ed949bf25be17bc"} Nov 02 08:40:52.420496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.420487 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:40:52.420919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.420867 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:52.533525 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:52.533461 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" podStartSLOduration=1.533440258 podStartE2EDuration="1.533440258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:52.528680768 +0000 UTC m=+2051.082184990" watchObservedRunningTime="2026-11-02 08:40:52.533440258 +0000 UTC m=+2051.086944460" Nov 02 08:40:53.431024 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:53.430988 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" event={"ID":"8a6b737b-8aeb-47e7-920a-33a1d396b983","Type":"ContainerStarted","Data":"0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc"} Nov 02 08:40:53.431524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:53.431053 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" event={"ID":"8a6b737b-8aeb-47e7-920a-33a1d396b983","Type":"ContainerStarted","Data":"123e0de7103f2a6df6350ca431ee913188d25b66e7bd98bd2fe51a7c48865f78"} Nov 02 08:40:53.431524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:53.431084 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:40:53.547375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:53.547313 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" podStartSLOduration=2.547298493 podStartE2EDuration="2.547298493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:53.543416174 +0000 UTC m=+2052.096920384" watchObservedRunningTime="2026-11-02 08:40:53.547298493 +0000 UTC m=+2052.100802689" Nov 02 08:40:53.584032 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:53.584001 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab5677e-be56-4ada-a457-7ee7d9ad4cda" path="/var/lib/kubelet/pods/7ab5677e-be56-4ada-a457-7ee7d9ad4cda/volumes" Nov 02 08:40:55.189747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:55.189695 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:55.361202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:55.361131 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:57.446420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:57.446375 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:58.004472 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:58.004436 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:58.222762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:58.222726 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:58.580140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:58.580107 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:58.777314 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:58.777274 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:58.910073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:58.909994 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t"] Nov 02 08:40:58.910253 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:58.910211 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" containerID="cri-o://7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f" gracePeriod=30 Nov 02 08:40:58.949409 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:58.949376 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:59.203249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:59.203212 6160 patch_prober.go:29] interesting pod/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:40:59.203441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:40:59.203288 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:41:01.208705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.208654 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:01.337742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.337712 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb"] Nov 02 08:41:01.343234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.343216 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.365941 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.365913 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.366091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.365962 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6fe81-2539-40b8-b08c-8ed5033da34c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.366091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.366015 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.366091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.366042 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh9xb\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-kube-api-access-lh9xb\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.366247 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.366143 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.366247 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.366187 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.366327 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.366278 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.366327 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.366322 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.366435 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.366344 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.424663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.424611 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb"] Nov 02 08:41:01.467784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.467691 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6fe81-2539-40b8-b08c-8ed5033da34c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.467784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.467737 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.467784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.467774 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh9xb\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-kube-api-access-lh9xb\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468077 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.467932 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468077 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.467984 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.468076 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.468139 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.468189 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.468277 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468450 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.468382 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468514 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.468497 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468571 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.468525 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6fe81-2539-40b8-b08c-8ed5033da34c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.468674 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.468863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.468842 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.470828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.470768 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.471039 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.471021 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.521741 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.521703 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh9xb\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-kube-api-access-lh9xb\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.521903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.521770 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-x8nhb\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.656547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.656513 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:01.685691 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:01.685660 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:02.081833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.081805 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb"] Nov 02 08:41:02.082754 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:41:02.082727 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d6fe81_2539_40b8_b08c_8ed5033da34c.slice/crio-5cf156bb24f351186a112420dd64059df92505d8901bf350f8b1dd7383c2e2b7 WatchSource:0}: Error finding container 5cf156bb24f351186a112420dd64059df92505d8901bf350f8b1dd7383c2e2b7: Status 404 returned error can't find the container with id 5cf156bb24f351186a112420dd64059df92505d8901bf350f8b1dd7383c2e2b7 Nov 02 08:41:02.085280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.085247 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:41:02.085372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.085311 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:41:02.085372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.085339 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:41:02.507005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.506948 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" event={"ID":"20d6fe81-2539-40b8-b08c-8ed5033da34c","Type":"ContainerStarted","Data":"fbab1e84bb3104efdea65a303a0541159ce5d24436f5d032235734fede1d7810"} Nov 02 08:41:02.507520 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.507019 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" event={"ID":"20d6fe81-2539-40b8-b08c-8ed5033da34c","Type":"ContainerStarted","Data":"5cf156bb24f351186a112420dd64059df92505d8901bf350f8b1dd7383c2e2b7"} Nov 02 08:41:02.626502 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.626440 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" podStartSLOduration=1.626420119 podStartE2EDuration="1.626420119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:02.625750815 +0000 UTC m=+2061.179255040" watchObservedRunningTime="2026-11-02 08:41:02.626420119 +0000 UTC m=+2061.179924319" Nov 02 08:41:02.657125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.657085 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:02.770956 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.770871 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf"] Nov 02 08:41:02.773817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.773799 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:02.783405 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.783195 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkkp\" (UniqueName: \"kubernetes.io/projected/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-kube-api-access-9zkkp\") pod \"payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf\" (UID: \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:02.783549 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.783504 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf\" (UID: \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:02.832460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.832423 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d51175\"" Nov 02 08:41:02.832672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.832558 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d51175-dockercfg-tpd8s\"" Nov 02 08:41:02.836429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.836395 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf"] Nov 02 08:41:02.884184 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.884140 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkkp\" (UniqueName: \"kubernetes.io/projected/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-kube-api-access-9zkkp\") pod \"payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf\" (UID: \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:02.884349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.884272 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf\" (UID: \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:02.884943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.884922 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf\" (UID: \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:02.940620 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:02.940567 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkkp\" (UniqueName: \"kubernetes.io/projected/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-kube-api-access-9zkkp\") pod \"payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf\" (UID: \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:03.089495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.089466 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:03.434544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.434508 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:41:03.492723 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:41:03.492695 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb4d9c6_5abb_4536_a36e_50390e6dabf0.slice/crio-d6e85893c7964e08b188e9d3dee068fafec0b06db4d80a1f3bc17d236539d75a WatchSource:0}: Error finding container d6e85893c7964e08b188e9d3dee068fafec0b06db4d80a1f3bc17d236539d75a: Status 404 returned error can't find the container with id d6e85893c7964e08b188e9d3dee068fafec0b06db4d80a1f3bc17d236539d75a Nov 02 08:41:03.535663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.535629 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf"] Nov 02 08:41:03.538249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.538220 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" event={"ID":"5bb4d9c6-5abb-4536-a36e-50390e6dabf0","Type":"ContainerStarted","Data":"d6e85893c7964e08b188e9d3dee068fafec0b06db4d80a1f3bc17d236539d75a"} Nov 02 08:41:03.539758 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.539733 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr"] Nov 02 08:41:03.543904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.543882 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:03.594377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.592699 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fxz\" (UniqueName: \"kubernetes.io/projected/a9d93ffc-c954-4760-8606-29cf9767c36b-kube-api-access-x7fxz\") pod \"payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr\" (UID: \"a9d93ffc-c954-4760-8606-29cf9767c36b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:03.594377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.592891 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d93ffc-c954-4760-8606-29cf9767c36b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr\" (UID: \"a9d93ffc-c954-4760-8606-29cf9767c36b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:03.594996 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.594974 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr"] Nov 02 08:41:03.659393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.658118 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-x8nhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:41:03.659393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.658213 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 08:41:03.694330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.694241 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fxz\" (UniqueName: \"kubernetes.io/projected/a9d93ffc-c954-4760-8606-29cf9767c36b-kube-api-access-x7fxz\") pod \"payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr\" (UID: \"a9d93ffc-c954-4760-8606-29cf9767c36b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:03.694490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.694341 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d93ffc-c954-4760-8606-29cf9767c36b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr\" (UID: \"a9d93ffc-c954-4760-8606-29cf9767c36b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:03.695094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.695073 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d93ffc-c954-4760-8606-29cf9767c36b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr\" (UID: \"a9d93ffc-c954-4760-8606-29cf9767c36b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:03.748811 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.748776 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fxz\" (UniqueName: \"kubernetes.io/projected/a9d93ffc-c954-4760-8606-29cf9767c36b-kube-api-access-x7fxz\") pod \"payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr\" (UID: \"a9d93ffc-c954-4760-8606-29cf9767c36b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:03.896950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:03.896919 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:04.245005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.244980 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:41:04.282703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.282677 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr"] Nov 02 08:41:04.284431 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:41:04.284407 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d93ffc_c954_4760_8606_29cf9767c36b.slice/crio-f0e7539f4fb79e01ccaacb81ebef6742ea6ee0fa1a04e0f34604f66e393d5cfe WatchSource:0}: Error finding container f0e7539f4fb79e01ccaacb81ebef6742ea6ee0fa1a04e0f34604f66e393d5cfe: Status 404 returned error can't find the container with id f0e7539f4fb79e01ccaacb81ebef6742ea6ee0fa1a04e0f34604f66e393d5cfe Nov 02 08:41:04.301670 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.301649 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a90a9e-68c6-44e6-b25f-4697524795f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a90a9e-68c6-44e6-b25f-4697524795f6-istiod-ca-cert\") pod \"95a90a9e-68c6-44e6-b25f-4697524795f6\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " Nov 02 08:41:04.301749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.301687 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-podinfo\") pod \"95a90a9e-68c6-44e6-b25f-4697524795f6\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " Nov 02 08:41:04.301749 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.301731 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-kube-api-access-htn6k\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-kube-api-access-htn6k\") pod \"95a90a9e-68c6-44e6-b25f-4697524795f6\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " Nov 02 08:41:04.301822 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.301763 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-data\") pod \"95a90a9e-68c6-44e6-b25f-4697524795f6\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " Nov 02 08:41:04.301874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.301859 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-credential-socket\") pod \"95a90a9e-68c6-44e6-b25f-4697524795f6\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " Nov 02 08:41:04.301964 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.301947 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-certs\") pod \"95a90a9e-68c6-44e6-b25f-4697524795f6\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " Nov 02 08:41:04.302024 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.301998 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-token\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-token\") pod \"95a90a9e-68c6-44e6-b25f-4697524795f6\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " Nov 02 08:41:04.302079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302037 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-socket\") pod \"95a90a9e-68c6-44e6-b25f-4697524795f6\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " Nov 02 08:41:04.302134 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302092 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a90a9e-68c6-44e6-b25f-4697524795f6-istiod-ca-cert" pod "95a90a9e-68c6-44e6-b25f-4697524795f6" (UID: "95a90a9e-68c6-44e6-b25f-4697524795f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:41:04.302134 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302104 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-envoy\") pod \"95a90a9e-68c6-44e6-b25f-4697524795f6\" (UID: \"95a90a9e-68c6-44e6-b25f-4697524795f6\") " Nov 02 08:41:04.302302 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302280 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-credential-socket" pod "95a90a9e-68c6-44e6-b25f-4697524795f6" (UID: "95a90a9e-68c6-44e6-b25f-4697524795f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.302479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302388 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-data" pod "95a90a9e-68c6-44e6-b25f-4697524795f6" (UID: "95a90a9e-68c6-44e6-b25f-4697524795f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.302479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302477 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-socket" pod "95a90a9e-68c6-44e6-b25f-4697524795f6" (UID: "95a90a9e-68c6-44e6-b25f-4697524795f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.302734 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302712 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-certs" pod "95a90a9e-68c6-44e6-b25f-4697524795f6" (UID: "95a90a9e-68c6-44e6-b25f-4697524795f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.302734 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302722 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.302854 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302747 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a90a9e-68c6-44e6-b25f-4697524795f6-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.302854 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302763 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.302854 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.302777 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.305466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.305410 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-podinfo" pod "95a90a9e-68c6-44e6-b25f-4697524795f6" (UID: "95a90a9e-68c6-44e6-b25f-4697524795f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:41:04.305466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.305421 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-envoy" pod "95a90a9e-68c6-44e6-b25f-4697524795f6" (UID: "95a90a9e-68c6-44e6-b25f-4697524795f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:04.306879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.306048 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-token" pod "95a90a9e-68c6-44e6-b25f-4697524795f6" (UID: "95a90a9e-68c6-44e6-b25f-4697524795f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:04.339201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.338501 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-kube-api-access-htn6k" pod "95a90a9e-68c6-44e6-b25f-4697524795f6" (UID: "95a90a9e-68c6-44e6-b25f-4697524795f6"). InnerVolumeSpecName "kube-api-access-htn6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:04.403765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.403733 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.403765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.403764 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.404334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.403774 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.404334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.403783 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a90a9e-68c6-44e6-b25f-4697524795f6-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.404334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.403792 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htn6k\" (UniqueName: \"kubernetes.io/projected/95a90a9e-68c6-44e6-b25f-4697524795f6-kube-api-access-htn6k\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:04.441223 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.440946 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:41:04.549031 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.548683 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" event={"ID":"5bb4d9c6-5abb-4536-a36e-50390e6dabf0","Type":"ContainerStarted","Data":"fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56"} Nov 02 08:41:04.550129 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.549686 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:04.552584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.551996 6160 generic.go:356] "Generic (PLEG): container finished" podID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerID="7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f" exitCode=0 Nov 02 08:41:04.552584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.552065 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" Nov 02 08:41:04.552584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.552077 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" event={"ID":"95a90a9e-68c6-44e6-b25f-4697524795f6","Type":"ContainerDied","Data":"7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f"} Nov 02 08:41:04.552584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.552133 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t" event={"ID":"95a90a9e-68c6-44e6-b25f-4697524795f6","Type":"ContainerDied","Data":"999c336a7acb4b5f3c7870ce57fa748edf700eb8f1783b09fbe84830e94a052e"} Nov 02 08:41:04.552584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.552163 6160 scope.go:122] "RemoveContainer" containerID="7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f" Nov 02 08:41:04.557004 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.556981 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" event={"ID":"a9d93ffc-c954-4760-8606-29cf9767c36b","Type":"ContainerStarted","Data":"bbebd5c977381981e40070b08f86c66eb5f20483b2afe28bb6db5bba607273bd"} Nov 02 08:41:04.557107 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.557026 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" event={"ID":"a9d93ffc-c954-4760-8606-29cf9767c36b","Type":"ContainerStarted","Data":"f0e7539f4fb79e01ccaacb81ebef6742ea6ee0fa1a04e0f34604f66e393d5cfe"} Nov 02 08:41:04.557107 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.557054 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:04.567997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.567945 6160 scope.go:122] "RemoveContainer" containerID="7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f" Nov 02 08:41:04.568257 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:41:04.568237 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f\": container with ID starting with 7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f not found: ID does not exist" containerID="7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f" Nov 02 08:41:04.568359 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.568262 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f"} err="failed to get container status \"7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f\": rpc error: code = NotFound desc = could not find container \"7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f\": container with ID starting with 7689e84dbb475e8cb38ce4d7655233c12b5bbe9499f89f46b3571f980196686f not found: ID does not exist" Nov 02 08:41:04.662259 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.660323 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" podStartSLOduration=2.660301651 podStartE2EDuration="2.660301651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:04.658581994 +0000 UTC m=+2063.212086210" watchObservedRunningTime="2026-11-02 08:41:04.660301651 +0000 UTC m=+2063.213805849" Nov 02 08:41:04.662586 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.662561 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-x8nhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:41:04.662677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.662611 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 08:41:04.790849 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.790769 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t"] Nov 02 08:41:04.795490 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.795463 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-69a9bb-openshift-default-54c485cdc7-zmg2t"] Nov 02 08:41:04.883093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.883056 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:04.901694 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:04.901642 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" podStartSLOduration=1.901628858 podStartE2EDuration="1.901628858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:04.897805754 +0000 UTC m=+2063.451309946" watchObservedRunningTime="2026-11-02 08:41:04.901628858 +0000 UTC m=+2063.455133055" Nov 02 08:41:05.585052 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:05.585017 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" path="/var/lib/kubelet/pods/95a90a9e-68c6-44e6-b25f-4697524795f6/volumes" Nov 02 08:41:05.657215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:05.657185 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-x8nhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:41:05.657393 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:05.657236 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 08:41:06.658132 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:06.658100 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-x8nhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:41:06.658618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:06.658154 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 08:41:07.630268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:07.630199 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:07.657271 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:07.657240 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-x8nhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:41:07.657475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:07.657297 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 08:41:08.657129 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:08.657093 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-x8nhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:41:08.657548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:08.657151 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 08:41:09.657257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:09.657225 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-x8nhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 08:41:09.657720 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:09.657286 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 08:41:10.521543 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:41:10.521506 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:41:10.661503 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:10.661468 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:10.661932 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:10.661750 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:10.662372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:10.662355 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:41:11.739812 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:11.739772 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:15.566407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:15.566373 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:41:15.566892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:15.566458 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:41:16.534984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:16.534925 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:19.795962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:19.795925 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:20.478505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:20.478457 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:20.736246 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:20.736131 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7"] Nov 02 08:41:20.736448 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:20.736399 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" podUID="08381029-9e19-447b-93cb-7722244cad27" containerName="maas-api" containerID="cri-o://f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d" gracePeriod=30 Nov 02 08:41:20.882969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:20.882928 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:20.884944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:20.884908 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" podUID="08381029-9e19-447b-93cb-7722244cad27" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.70:8443/health\": dial tcp 10.133.0.70:8443: connect: connection refused" Nov 02 08:41:21.005091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.005024 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:41:21.131391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.131355 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls\") pod \"08381029-9e19-447b-93cb-7722244cad27\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " Nov 02 08:41:21.131604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.131434 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-metrics-certs\") pod \"08381029-9e19-447b-93cb-7722244cad27\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " Nov 02 08:41:21.131604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.131510 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08381029-9e19-447b-93cb-7722244cad27-kube-api-access-ngrrx\" (UniqueName: \"kubernetes.io/projected/08381029-9e19-447b-93cb-7722244cad27-kube-api-access-ngrrx\") pod \"08381029-9e19-447b-93cb-7722244cad27\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " Nov 02 08:41:21.131604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.131546 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08381029-9e19-447b-93cb-7722244cad27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08381029-9e19-447b-93cb-7722244cad27-openshift-service-ca\") pod \"08381029-9e19-447b-93cb-7722244cad27\" (UID: \"08381029-9e19-447b-93cb-7722244cad27\") " Nov 02 08:41:21.132246 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.132155 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08381029-9e19-447b-93cb-7722244cad27-openshift-service-ca" pod "08381029-9e19-447b-93cb-7722244cad27" (UID: "08381029-9e19-447b-93cb-7722244cad27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:41:21.133649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.133609 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:21.134435 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.134412 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-metrics-certs" pod "08381029-9e19-447b-93cb-7722244cad27" (UID: "08381029-9e19-447b-93cb-7722244cad27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:41:21.134623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.134438 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls" pod "08381029-9e19-447b-93cb-7722244cad27" (UID: "08381029-9e19-447b-93cb-7722244cad27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:41:21.134623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.134578 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08381029-9e19-447b-93cb-7722244cad27-kube-api-access-ngrrx" pod "08381029-9e19-447b-93cb-7722244cad27" (UID: "08381029-9e19-447b-93cb-7722244cad27"). InnerVolumeSpecName "kube-api-access-ngrrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:21.232820 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.232788 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:21.232820 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.232816 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08381029-9e19-447b-93cb-7722244cad27-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:21.232820 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.232826 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngrrx\" (UniqueName: \"kubernetes.io/projected/08381029-9e19-447b-93cb-7722244cad27-kube-api-access-ngrrx\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:21.233055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.232837 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08381029-9e19-447b-93cb-7722244cad27-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:21.673616 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.673578 6160 generic.go:356] "Generic (PLEG): container finished" podID="08381029-9e19-447b-93cb-7722244cad27" containerID="f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d" exitCode=0 Nov 02 08:41:21.673817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.673627 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" Nov 02 08:41:21.673817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.673668 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" event={"ID":"08381029-9e19-447b-93cb-7722244cad27","Type":"ContainerDied","Data":"f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d"} Nov 02 08:41:21.673817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.673720 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7" event={"ID":"08381029-9e19-447b-93cb-7722244cad27","Type":"ContainerDied","Data":"573da68012da4a7b46b853f02c9759fc4676db696b95e0cbe907575b79096551"} Nov 02 08:41:21.673817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.673747 6160 scope.go:122] "RemoveContainer" containerID="f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d" Nov 02 08:41:21.684088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.684069 6160 scope.go:122] "RemoveContainer" containerID="f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d" Nov 02 08:41:21.684463 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:41:21.684438 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d\": container with ID starting with f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d not found: ID does not exist" containerID="f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d" Nov 02 08:41:21.684573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.684469 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d"} err="failed to get container status \"f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d\": rpc error: code = NotFound desc = could not find container \"f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d\": container with ID starting with f9ef3e401dfbdd257fbcb5f282cab5eeb01f72157852b87ffa700d1097c00d2d not found: ID does not exist" Nov 02 08:41:21.829133 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.829074 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7"] Nov 02 08:41:21.835244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:21.833606 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d77b6-554f94d8c-5mlz7"] Nov 02 08:41:22.523990 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:22.523955 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:22.719394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:22.719354 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:22.880938 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:22.880845 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:23.448101 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:23.448071 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:23.586831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:23.586796 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08381029-9e19-447b-93cb-7722244cad27" path="/var/lib/kubelet/pods/08381029-9e19-447b-93cb-7722244cad27/volumes" Nov 02 08:41:24.257446 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:24.257401 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:25.198703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:25.198658 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:25.428870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:25.428828 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:27.367207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.367155 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd"] Nov 02 08:41:27.368418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.368387 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" podUID="8a7f6ab0-5386-419a-b710-182ac04dd9c2" containerName="maas-api" containerID="cri-o://6054f1606f9bd6b42ad4ce2743569536039fa55217312229ae24f9a7d0a708d9" gracePeriod=30 Nov 02 08:41:27.495881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.495704 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:27.775205 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.773482 6160 generic.go:356] "Generic (PLEG): container finished" podID="8a7f6ab0-5386-419a-b710-182ac04dd9c2" containerID="6054f1606f9bd6b42ad4ce2743569536039fa55217312229ae24f9a7d0a708d9" exitCode=0 Nov 02 08:41:27.775205 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.773539 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" event={"ID":"8a7f6ab0-5386-419a-b710-182ac04dd9c2","Type":"ContainerDied","Data":"6054f1606f9bd6b42ad4ce2743569536039fa55217312229ae24f9a7d0a708d9"} Nov 02 08:41:27.775205 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.773589 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" event={"ID":"8a7f6ab0-5386-419a-b710-182ac04dd9c2","Type":"ContainerDied","Data":"a3185c5cdbc48b825d70207fa0b9b87e23b4096d2cc56f0f575bd184bb52bc15"} Nov 02 08:41:27.775205 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.773614 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3185c5cdbc48b825d70207fa0b9b87e23b4096d2cc56f0f575bd184bb52bc15" Nov 02 08:41:27.775205 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.775014 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:41:27.835201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.834473 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-metrics-certs\") pod \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " Nov 02 08:41:27.835201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.834659 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7f6ab0-5386-419a-b710-182ac04dd9c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7f6ab0-5386-419a-b710-182ac04dd9c2-openshift-service-ca\") pod \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " Nov 02 08:41:27.835201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.834754 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7f6ab0-5386-419a-b710-182ac04dd9c2-kube-api-access-6fd97\" (UniqueName: \"kubernetes.io/projected/8a7f6ab0-5386-419a-b710-182ac04dd9c2-kube-api-access-6fd97\") pod \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " Nov 02 08:41:27.835201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.834813 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-maas-api-tls\") pod \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\" (UID: \"8a7f6ab0-5386-419a-b710-182ac04dd9c2\") " Nov 02 08:41:27.836501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.836231 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7f6ab0-5386-419a-b710-182ac04dd9c2-openshift-service-ca" pod "8a7f6ab0-5386-419a-b710-182ac04dd9c2" (UID: "8a7f6ab0-5386-419a-b710-182ac04dd9c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:41:27.837709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.837686 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7f6ab0-5386-419a-b710-182ac04dd9c2-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:27.858200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.856446 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-metrics-certs" pod "8a7f6ab0-5386-419a-b710-182ac04dd9c2" (UID: "8a7f6ab0-5386-419a-b710-182ac04dd9c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:41:27.858200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.856566 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-maas-api-tls" pod "8a7f6ab0-5386-419a-b710-182ac04dd9c2" (UID: "8a7f6ab0-5386-419a-b710-182ac04dd9c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:41:27.858586 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.858545 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7f6ab0-5386-419a-b710-182ac04dd9c2-kube-api-access-6fd97" pod "8a7f6ab0-5386-419a-b710-182ac04dd9c2" (UID: "8a7f6ab0-5386-419a-b710-182ac04dd9c2"). InnerVolumeSpecName "kube-api-access-6fd97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:27.952722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.952686 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fd97\" (UniqueName: \"kubernetes.io/projected/8a7f6ab0-5386-419a-b710-182ac04dd9c2-kube-api-access-6fd97\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:27.952967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.952948 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:27.953084 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:27.953070 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7f6ab0-5386-419a-b710-182ac04dd9c2-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:28.485908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.485874 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:28.700199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.699724 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:28.780907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.780878 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd" Nov 02 08:41:28.938290 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.938255 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5"] Nov 02 08:41:28.939147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.939123 6160 cpu_manager.go:414] "containerMap: removing container" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" Nov 02 08:41:28.939261 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.939150 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" Nov 02 08:41:28.939261 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.939215 6160 cpu_manager.go:414] "containerMap: removing container" podUID="8a7f6ab0-5386-419a-b710-182ac04dd9c2" containerName="maas-api" Nov 02 08:41:28.939261 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.939225 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7f6ab0-5386-419a-b710-182ac04dd9c2" containerName="maas-api" Nov 02 08:41:28.939261 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.939256 6160 cpu_manager.go:414] "containerMap: removing container" podUID="08381029-9e19-447b-93cb-7722244cad27" containerName="maas-api" Nov 02 08:41:28.939494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.939263 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08381029-9e19-447b-93cb-7722244cad27" containerName="maas-api" Nov 02 08:41:28.939494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.939400 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="08381029-9e19-447b-93cb-7722244cad27" containerName="maas-api" Nov 02 08:41:28.939494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.939415 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a90a9e-68c6-44e6-b25f-4697524795f6" containerName="istio-proxy" Nov 02 08:41:28.939494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.939430 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7f6ab0-5386-419a-b710-182ac04dd9c2" containerName="maas-api" Nov 02 08:41:28.942471 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.942448 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:28.958319 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.958249 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd"] Nov 02 08:41:28.962143 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.962119 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-108cf9-55446f7859-9hxcd"] Nov 02 08:41:28.991878 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.991846 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d5a5a-dockercfg-h6r46\"" Nov 02 08:41:28.992110 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.991969 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d5a5a\"" Nov 02 08:41:28.995336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:28.995310 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5"] Nov 02 08:41:29.067374 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.067339 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j9wh\" (UniqueName: \"kubernetes.io/projected/8a4f73f1-67e3-4516-9dd7-961ff50118e3-kube-api-access-9j9wh\") pod \"payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5\" (UID: \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:29.067551 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.067403 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4f73f1-67e3-4516-9dd7-961ff50118e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5\" (UID: \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:29.112900 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.112863 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d"] Nov 02 08:41:29.115596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.115579 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:29.169090 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.169061 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j9wh\" (UniqueName: \"kubernetes.io/projected/8a4f73f1-67e3-4516-9dd7-961ff50118e3-kube-api-access-9j9wh\") pod \"payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5\" (UID: \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:29.169277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.169118 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4f73f1-67e3-4516-9dd7-961ff50118e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5\" (UID: \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:29.169949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.169920 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4f73f1-67e3-4516-9dd7-961ff50118e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5\" (UID: \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:29.171758 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.171737 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d"] Nov 02 08:41:29.224639 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.224565 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j9wh\" (UniqueName: \"kubernetes.io/projected/8a4f73f1-67e3-4516-9dd7-961ff50118e3-kube-api-access-9j9wh\") pod \"payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5\" (UID: \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:29.259229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.259143 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:29.270952 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.270920 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdr5h\" (UniqueName: \"kubernetes.io/projected/777b2e05-d870-4293-84f2-f166f7e44667-kube-api-access-hdr5h\") pod \"payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d\" (UID: \"777b2e05-d870-4293-84f2-f166f7e44667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:29.271137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.271117 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b2e05-d870-4293-84f2-f166f7e44667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d\" (UID: \"777b2e05-d870-4293-84f2-f166f7e44667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:29.372223 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.372165 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdr5h\" (UniqueName: \"kubernetes.io/projected/777b2e05-d870-4293-84f2-f166f7e44667-kube-api-access-hdr5h\") pod \"payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d\" (UID: \"777b2e05-d870-4293-84f2-f166f7e44667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:29.372363 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.372291 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b2e05-d870-4293-84f2-f166f7e44667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d\" (UID: \"777b2e05-d870-4293-84f2-f166f7e44667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:29.372932 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.372914 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b2e05-d870-4293-84f2-f166f7e44667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d\" (UID: \"777b2e05-d870-4293-84f2-f166f7e44667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:29.426755 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.426722 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdr5h\" (UniqueName: \"kubernetes.io/projected/777b2e05-d870-4293-84f2-f166f7e44667-kube-api-access-hdr5h\") pod \"payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d\" (UID: \"777b2e05-d870-4293-84f2-f166f7e44667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:29.427498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.427479 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:29.584346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.584315 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7f6ab0-5386-419a-b710-182ac04dd9c2" path="/var/lib/kubelet/pods/8a7f6ab0-5386-419a-b710-182ac04dd9c2/volumes" Nov 02 08:41:29.637919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.637888 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5"] Nov 02 08:41:29.638505 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:41:29.638482 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4f73f1_67e3_4516_9dd7_961ff50118e3.slice/crio-28feda14af4231d508c32ba2e27cdac4759487f8a79938fd15086898f3a3b5e3 WatchSource:0}: Error finding container 28feda14af4231d508c32ba2e27cdac4759487f8a79938fd15086898f3a3b5e3: Status 404 returned error can't find the container with id 28feda14af4231d508c32ba2e27cdac4759487f8a79938fd15086898f3a3b5e3 Nov 02 08:41:29.651608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.651580 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:29.790591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.790493 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" event={"ID":"8a4f73f1-67e3-4516-9dd7-961ff50118e3","Type":"ContainerStarted","Data":"5e6abf256f2fcd48d55e89156cf851e8fcacf6bd02d3ae755a34cc0c8bb3b2ce"} Nov 02 08:41:29.790591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.790557 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" event={"ID":"8a4f73f1-67e3-4516-9dd7-961ff50118e3","Type":"ContainerStarted","Data":"28feda14af4231d508c32ba2e27cdac4759487f8a79938fd15086898f3a3b5e3"} Nov 02 08:41:29.790591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.790584 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:29.798036 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.798012 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d"] Nov 02 08:41:29.798789 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:41:29.798769 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777b2e05_d870_4293_84f2_f166f7e44667.slice/crio-7737d6d6feb4b0cacc90acbdaa079b12df7546b2200918c220d36e8c4c58d690 WatchSource:0}: Error finding container 7737d6d6feb4b0cacc90acbdaa079b12df7546b2200918c220d36e8c4c58d690: Status 404 returned error can't find the container with id 7737d6d6feb4b0cacc90acbdaa079b12df7546b2200918c220d36e8c4c58d690 Nov 02 08:41:29.869492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.869434 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:29.901503 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:29.901457 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" podStartSLOduration=1.901443493 podStartE2EDuration="1.901443493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:29.898871337 +0000 UTC m=+2088.452375534" watchObservedRunningTime="2026-11-02 08:41:29.901443493 +0000 UTC m=+2088.454947690" Nov 02 08:41:30.036606 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:30.036567 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:30.216590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:30.216553 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:30.577783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:30.577692 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:30.799365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:30.799331 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" event={"ID":"777b2e05-d870-4293-84f2-f166f7e44667","Type":"ContainerStarted","Data":"12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a"} Nov 02 08:41:30.799878 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:30.799383 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" event={"ID":"777b2e05-d870-4293-84f2-f166f7e44667","Type":"ContainerStarted","Data":"7737d6d6feb4b0cacc90acbdaa079b12df7546b2200918c220d36e8c4c58d690"} Nov 02 08:41:30.799878 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:30.799417 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:30.906809 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:30.906699 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" podStartSLOduration=1.9066816439999998 podStartE2EDuration="1.906681644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:30.905565438 +0000 UTC m=+2089.459069657" watchObservedRunningTime="2026-11-02 08:41:30.906681644 +0000 UTC m=+2089.460185841" Nov 02 08:41:34.922343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:34.922307 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:36.506367 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:36.506331 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:36.675812 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:36.675778 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:36.806948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:36.806854 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77"] Nov 02 08:41:36.807149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:36.807101 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" containerID="cri-o://e5fc1c7f57085fe6e75e6f5ed62b7b1c76e04ef956286497306fde2d90445095" gracePeriod=30 Nov 02 08:41:36.871072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:36.871037 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:39.867271 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:39.867231 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:40.522430 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:41:40.522393 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:41:40.803725 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:40.803644 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:41:41.809376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:41.809339 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:41:41.946654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:41.946623 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:41:41.955083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:41.955056 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:41:42.097221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.097197 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:41:42.116949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.116923 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1cca66b-c26c-48f6-acaa-6c44537a259e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cca66b-c26c-48f6-acaa-6c44537a259e-istiod-ca-cert\") pod \"f1cca66b-c26c-48f6-acaa-6c44537a259e\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " Nov 02 08:41:42.117050 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.116997 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-kube-api-access-vsmpq\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-kube-api-access-vsmpq\") pod \"f1cca66b-c26c-48f6-acaa-6c44537a259e\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " Nov 02 08:41:42.117099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117063 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-envoy\") pod \"f1cca66b-c26c-48f6-acaa-6c44537a259e\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " Nov 02 08:41:42.117144 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117112 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-token\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-token\") pod \"f1cca66b-c26c-48f6-acaa-6c44537a259e\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " Nov 02 08:41:42.117256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117238 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-podinfo\") pod \"f1cca66b-c26c-48f6-acaa-6c44537a259e\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " Nov 02 08:41:42.117321 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117280 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-credential-socket\") pod \"f1cca66b-c26c-48f6-acaa-6c44537a259e\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " Nov 02 08:41:42.117386 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117318 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-data\") pod \"f1cca66b-c26c-48f6-acaa-6c44537a259e\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " Nov 02 08:41:42.117386 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117340 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1cca66b-c26c-48f6-acaa-6c44537a259e-istiod-ca-cert" pod "f1cca66b-c26c-48f6-acaa-6c44537a259e" (UID: "f1cca66b-c26c-48f6-acaa-6c44537a259e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:41:42.117523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117441 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-socket\") pod \"f1cca66b-c26c-48f6-acaa-6c44537a259e\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " Nov 02 08:41:42.117653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117594 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-credential-socket" pod "f1cca66b-c26c-48f6-acaa-6c44537a259e" (UID: "f1cca66b-c26c-48f6-acaa-6c44537a259e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:42.117653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117618 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-data" pod "f1cca66b-c26c-48f6-acaa-6c44537a259e" (UID: "f1cca66b-c26c-48f6-acaa-6c44537a259e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:42.117901 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117649 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-certs\") pod \"f1cca66b-c26c-48f6-acaa-6c44537a259e\" (UID: \"f1cca66b-c26c-48f6-acaa-6c44537a259e\") " Nov 02 08:41:42.117901 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.117758 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-socket" pod "f1cca66b-c26c-48f6-acaa-6c44537a259e" (UID: "f1cca66b-c26c-48f6-acaa-6c44537a259e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:42.118085 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.118070 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:42.118150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.118092 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:42.118150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.118102 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:42.118150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.118110 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cca66b-c26c-48f6-acaa-6c44537a259e-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:42.118150 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.118121 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-certs" pod "f1cca66b-c26c-48f6-acaa-6c44537a259e" (UID: "f1cca66b-c26c-48f6-acaa-6c44537a259e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:42.119547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.119526 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-token" pod "f1cca66b-c26c-48f6-acaa-6c44537a259e" (UID: "f1cca66b-c26c-48f6-acaa-6c44537a259e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:42.119674 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.119624 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-envoy" pod "f1cca66b-c26c-48f6-acaa-6c44537a259e" (UID: "f1cca66b-c26c-48f6-acaa-6c44537a259e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:41:42.119959 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.119940 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-podinfo" pod "f1cca66b-c26c-48f6-acaa-6c44537a259e" (UID: "f1cca66b-c26c-48f6-acaa-6c44537a259e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:41:42.120009 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.119954 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-kube-api-access-vsmpq" pod "f1cca66b-c26c-48f6-acaa-6c44537a259e" (UID: "f1cca66b-c26c-48f6-acaa-6c44537a259e"). InnerVolumeSpecName "kube-api-access-vsmpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:42.219987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.219945 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsmpq\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-kube-api-access-vsmpq\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:42.219987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.219985 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:42.220273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.219999 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:42.220273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.220011 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cca66b-c26c-48f6-acaa-6c44537a259e-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:42.220273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.220024 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cca66b-c26c-48f6-acaa-6c44537a259e-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:41:42.357719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.357639 6160 scope.go:122] "RemoveContainer" containerID="6054f1606f9bd6b42ad4ce2743569536039fa55217312229ae24f9a7d0a708d9" Nov 02 08:41:42.368769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.368747 6160 scope.go:122] "RemoveContainer" containerID="e5fc1c7f57085fe6e75e6f5ed62b7b1c76e04ef956286497306fde2d90445095" Nov 02 08:41:42.888927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.888895 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" Nov 02 08:41:42.889308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.888894 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" event={"ID":"f1cca66b-c26c-48f6-acaa-6c44537a259e","Type":"ContainerDied","Data":"e5fc1c7f57085fe6e75e6f5ed62b7b1c76e04ef956286497306fde2d90445095"} Nov 02 08:41:42.889308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:42.889032 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77" event={"ID":"f1cca66b-c26c-48f6-acaa-6c44537a259e","Type":"ContainerDied","Data":"f176971f6189a228539d62fe465b605116ad3bd494a856007575d21bf5d56e30"} Nov 02 08:41:43.102889 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:43.102853 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77"] Nov 02 08:41:43.105875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:43.105852 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-108cf9-openshift-default-7b9d95789-8dw77"] Nov 02 08:41:43.585208 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:43.585156 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" path="/var/lib/kubelet/pods/f1cca66b-c26c-48f6-acaa-6c44537a259e/volumes" Nov 02 08:41:44.645773 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:44.645735 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:46.656025 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.655988 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds"] Nov 02 08:41:46.656568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.656554 6160 cpu_manager.go:414] "containerMap: removing container" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" Nov 02 08:41:46.656631 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.656569 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" Nov 02 08:41:46.656688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.656679 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cca66b-c26c-48f6-acaa-6c44537a259e" containerName="istio-proxy" Nov 02 08:41:46.661861 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.661839 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.701918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.701887 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p"] Nov 02 08:41:46.705937 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.705919 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.757275 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.757232 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds"] Nov 02 08:41:46.767611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.767581 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.767810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.767631 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.767810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.767656 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.767810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.767803 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjph\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-kube-api-access-9hjph\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.767995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.767832 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.767995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.767854 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.767995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.767871 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73012162-12a0-467c-b849-1d9dd494c0b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.767995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.767935 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.767995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.767983 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.768297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.768001 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.768297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.768075 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.768297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.768100 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73012162-12a0-467c-b849-1d9dd494c0b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.768297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.768153 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.768297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.768196 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.768297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.768218 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.768297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.768243 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5l2v\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-kube-api-access-r5l2v\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.768297 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.768295 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.768797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.768337 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.805675 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.805635 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p"] Nov 02 08:41:46.869338 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869297 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5l2v\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-kube-api-access-r5l2v\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.869338 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869343 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.869651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869366 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.869651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869415 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.869651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869463 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.869651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869493 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.869651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869588 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjph\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-kube-api-access-9hjph\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.869651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869621 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.869651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869646 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869670 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73012162-12a0-467c-b849-1d9dd494c0b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869715 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869769 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869797 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869834 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869885 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869912 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73012162-12a0-467c-b849-1d9dd494c0b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869952 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.869985 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870010 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870109 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.870622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870146 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870396 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870479 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.870622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870600 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870888 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870771 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.870888 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870814 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73012162-12a0-467c-b849-1d9dd494c0b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870886 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.870984 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.870952 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.872305 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.872278 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.872489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.872468 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.872683 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.872661 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73012162-12a0-467c-b849-1d9dd494c0b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.872945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.872930 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.923352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.923310 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.923516 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.923377 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjph\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-kube-api-access-9hjph\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.924216 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.924164 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-rvn8p\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:46.924328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.924314 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5l2v\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-kube-api-access-r5l2v\") pod \"maas-default-gateway-openshift-default-769df7846c-mm2ds\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:46.978093 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:46.978062 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:47.019959 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.019923 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:47.400598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.400513 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds"] Nov 02 08:41:47.436926 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:41:47.436898 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec99e3d7_5c3d_4596_9b34_bdecbd94643f.slice/crio-1c843b9b4c0c959a2119bc8d2e4e10f274a9ed9450fd4bda957d10f6dae51485 WatchSource:0}: Error finding container 1c843b9b4c0c959a2119bc8d2e4e10f274a9ed9450fd4bda957d10f6dae51485: Status 404 returned error can't find the container with id 1c843b9b4c0c959a2119bc8d2e4e10f274a9ed9450fd4bda957d10f6dae51485 Nov 02 08:41:47.439524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.439481 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:41:47.439624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.439558 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:41:47.439624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.439593 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:41:47.456370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.456346 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p"] Nov 02 08:41:47.458283 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:41:47.458252 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73012162_12a0_467c_b849_1d9dd494c0b7.slice/crio-59e491a6252ef7cbd7bd4c1cf0a7c0e731f60ab5c2d8036e40de2336f7cbf6e6 WatchSource:0}: Error finding container 59e491a6252ef7cbd7bd4c1cf0a7c0e731f60ab5c2d8036e40de2336f7cbf6e6: Status 404 returned error can't find the container with id 59e491a6252ef7cbd7bd4c1cf0a7c0e731f60ab5c2d8036e40de2336f7cbf6e6 Nov 02 08:41:47.461549 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.461503 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:41:47.461653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.461601 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:41:47.461707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.461648 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:41:47.931416 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.931374 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" event={"ID":"73012162-12a0-467c-b849-1d9dd494c0b7","Type":"ContainerStarted","Data":"2593ff2802b4db422d2a579b6cf51d278acfc8a63bd7fb5d9a3659bffa9c36f9"} Nov 02 08:41:47.931902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.931458 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" event={"ID":"73012162-12a0-467c-b849-1d9dd494c0b7","Type":"ContainerStarted","Data":"59e491a6252ef7cbd7bd4c1cf0a7c0e731f60ab5c2d8036e40de2336f7cbf6e6"} Nov 02 08:41:47.933342 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.933318 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" event={"ID":"ec99e3d7-5c3d-4596-9b34-bdecbd94643f","Type":"ContainerStarted","Data":"9786f4c8cf0c5eb45703ffa571e30e054aeb6fe1b7ba3b974f384e60817c8bd9"} Nov 02 08:41:47.933501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.933358 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" event={"ID":"ec99e3d7-5c3d-4596-9b34-bdecbd94643f","Type":"ContainerStarted","Data":"1c843b9b4c0c959a2119bc8d2e4e10f274a9ed9450fd4bda957d10f6dae51485"} Nov 02 08:41:47.978833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:47.978765 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:48.024200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:48.021742 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:48.027664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:48.025324 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rvn8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:41:48.027664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:48.025403 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 08:41:48.057201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:48.053804 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podStartSLOduration=2.053779122 podStartE2EDuration="2.053779122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:48.045750135 +0000 UTC m=+2106.599254333" watchObservedRunningTime="2026-11-02 08:41:48.053779122 +0000 UTC m=+2106.607283322" Nov 02 08:41:48.162335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:48.162224 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" podStartSLOduration=2.16216821 podStartE2EDuration="2.16216821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:48.161550482 +0000 UTC m=+2106.715054696" watchObservedRunningTime="2026-11-02 08:41:48.16216821 +0000 UTC m=+2106.715672408" Nov 02 08:41:48.980614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:48.980584 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-mm2ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 08:41:48.981047 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:48.981024 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 08:41:49.039147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:49.039108 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rvn8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:41:49.039497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:49.039463 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 08:41:49.163866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:49.163788 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:49.955628 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:49.955589 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:49.979334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:49.979303 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-mm2ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 08:41:49.979707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:49.979357 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 08:41:50.026216 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:50.023715 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rvn8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:41:50.026216 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:50.023777 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 08:41:50.126200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:50.126144 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:50.294618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:50.294518 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:50.978713 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:50.978684 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-mm2ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 08:41:50.978962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:50.978939 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 08:41:51.043328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:51.043289 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rvn8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:41:51.043920 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:51.043883 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 08:41:51.979024 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:51.978981 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-mm2ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 08:41:51.979287 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:51.979053 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 08:41:52.028203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:52.026622 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rvn8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:41:52.028203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:52.026703 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 08:41:52.980048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:52.980000 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-mm2ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 08:41:52.980539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:52.980065 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 08:41:53.021672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:53.021631 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rvn8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:41:53.021947 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:53.021919 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 08:41:53.979168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:53.979134 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-mm2ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 08:41:53.979360 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:53.979218 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 08:41:54.020791 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:54.020754 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rvn8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:41:54.021208 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:54.020820 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 08:41:54.979271 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:54.979227 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-mm2ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 08:41:54.979440 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:54.979298 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 08:41:55.020614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:55.020571 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rvn8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 08:41:55.020786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:55.020638 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 08:41:55.265082 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:55.264972 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:56.052311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:56.052280 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:56.052845 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:56.052825 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:56.053758 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:56.053736 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:41:56.061208 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:56.061163 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:56.061606 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:56.061563 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:56.062685 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:56.062662 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:41:59.767209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:59.767159 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:59.929069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:41:59.929034 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:00.130078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:00.129989 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:00.336501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:00.336463 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:03.046250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:03.046210 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:03.710668 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:03.710627 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:06.166411 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:06.166371 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:06.700460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:06.700416 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:06.855306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:06.855262 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:07.827917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:07.827887 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:08.040089 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:08.040038 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:08.185106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:08.185075 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:08.691194 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:08.691133 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:10.525049 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:42:10.525008 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:42:12.145578 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:12.145539 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:13.140066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:13.140028 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:14.413949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:14.413911 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:14.577849 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:14.577815 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:14.735045 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:14.734955 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:15.211606 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:15.211566 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:18.000573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:18.000535 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:18.241555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:18.241522 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:20.159033 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:20.158996 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb"] Nov 02 08:42:20.205191 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:20.205148 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj"] Nov 02 08:42:20.272037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:20.271995 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:23.225316 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:23.225275 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:23.487170 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:23.487061 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:23.644945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:23.644908 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:24.660670 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:24.660630 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:27.424473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:27.424433 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9"] Nov 02 08:42:27.809248 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:27.809126 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:28.008018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:28.007979 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:28.169039 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:28.168998 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:28.904860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:28.904820 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:32.425461 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:32.425395 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" podUID="e3f40e31-04e8-4217-9e47-c486d3a06d80" containerName="payload-processing" containerID="cri-o://47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1" gracePeriod=25 Nov 02 08:42:32.932543 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:32.932505 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:33.083365 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:33.083328 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" podUID="e3f40e31-04e8-4217-9e47-c486d3a06d80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:42:35.181294 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.181245 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" podUID="2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" containerName="main" containerID="cri-o://394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370" gracePeriod=45 Nov 02 08:42:35.225674 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.225640 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" podUID="5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" containerName="main" containerID="cri-o://b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09" gracePeriod=45 Nov 02 08:42:35.482976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.482948 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:42:35.518101 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.518078 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:42:35.611654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.611613 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-dshm\") pod \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " Nov 02 08:42:35.611848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.611671 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-kube-api-access-cwj79\" (UniqueName: \"kubernetes.io/projected/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-kube-api-access-cwj79\") pod \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " Nov 02 08:42:35.611848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.611722 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-home\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-home\") pod \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " Nov 02 08:42:35.611848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.611754 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-home\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-home\") pod \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " Nov 02 08:42:35.611848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.611796 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-kube-api-access-pmc7x\" (UniqueName: \"kubernetes.io/projected/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-kube-api-access-pmc7x\") pod \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " Nov 02 08:42:35.612075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.611925 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tls-certs\") pod \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " Nov 02 08:42:35.612075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.611980 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-dshm\") pod \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " Nov 02 08:42:35.612075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.612043 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tls-certs\") pod \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " Nov 02 08:42:35.612260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.612093 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tmp-dir\") pod \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " Nov 02 08:42:35.612260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.612140 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-model-cache\") pod \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\" (UID: \"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde\") " Nov 02 08:42:35.612260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.612229 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-model-cache\") pod \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " Nov 02 08:42:35.612417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.612263 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-home" pod "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" (UID: "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.612417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.612274 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-home" pod "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" (UID: "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.612417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.612272 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tmp-dir\") pod \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\" (UID: \"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2\") " Nov 02 08:42:35.612586 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.612492 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tmp-dir" pod "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" (UID: "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.612704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.612682 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tmp-dir" pod "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" (UID: "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.613074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.613048 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-model-cache" pod "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" (UID: "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.613299 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.613276 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-model-cache" pod "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" (UID: "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.613465 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.613445 6160 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-model-cache\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.613604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.613588 6160 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tmp-dir\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.613712 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.613700 6160 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-home\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.613807 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.613795 6160 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-home\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.613904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.613892 6160 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tmp-dir\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.614926 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.614894 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-kube-api-access-cwj79" pod "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" (UID: "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2"). InnerVolumeSpecName "kube-api-access-cwj79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:35.615168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.615051 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-dshm" pod "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" (UID: "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.615168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.615070 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tls-certs" pod "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" (UID: "5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:42:35.616599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.616563 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-kube-api-access-pmc7x" pod "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" (UID: "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde"). InnerVolumeSpecName "kube-api-access-pmc7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:35.616847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.616815 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tls-certs" pod "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" (UID: "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:42:35.617370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.617345 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-dshm" pod "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" (UID: "2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:42:35.716125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.715936 6160 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-tls-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.716125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.715974 6160 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-dshm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.716125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.715990 6160 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-tls-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.716125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.716005 6160 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-model-cache\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.716125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.716021 6160 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-dshm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.716125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.716035 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwj79\" (UniqueName: \"kubernetes.io/projected/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2-kube-api-access-cwj79\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:35.716125 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:35.716049 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmc7x\" (UniqueName: \"kubernetes.io/projected/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde-kube-api-access-pmc7x\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:36.288599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.288559 6160 generic.go:356] "Generic (PLEG): container finished" podID="2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" containerID="394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370" exitCode=0 Nov 02 08:42:36.288994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.288604 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" Nov 02 08:42:36.288994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.288652 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" event={"ID":"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde","Type":"ContainerDied","Data":"394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370"} Nov 02 08:42:36.288994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.288698 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" event={"ID":"2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde","Type":"ContainerDied","Data":"94c00c5fbd9dbbe70280f699c8343f756c2e1faf0974e9c03bfe43f5091a0093"} Nov 02 08:42:36.288994 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.288717 6160 scope.go:122] "RemoveContainer" containerID="394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370" Nov 02 08:42:36.290503 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.290477 6160 generic.go:356] "Generic (PLEG): container finished" podID="5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" containerID="b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09" exitCode=0 Nov 02 08:42:36.290612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.290521 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" event={"ID":"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2","Type":"ContainerDied","Data":"b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09"} Nov 02 08:42:36.290612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.290548 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" event={"ID":"5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2","Type":"ContainerDied","Data":"314d2e5c1a13e5607aa732391e7b655e3805971183bd7784b907e8b281ac9549"} Nov 02 08:42:36.290612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.290527 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj" Nov 02 08:42:36.301664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.301639 6160 scope.go:122] "RemoveContainer" containerID="394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370" Nov 02 08:42:36.301985 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:42:36.301957 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370\": container with ID starting with 394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370 not found: ID does not exist" containerID="394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370" Nov 02 08:42:36.302081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.301995 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370"} err="failed to get container status \"394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370\": rpc error: code = NotFound desc = could not find container \"394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370\": container with ID starting with 394903208b5dab9fdcdadb748acaadd7ed3ac31c624eea8ec9b12e94afaf5370 not found: ID does not exist" Nov 02 08:42:36.302081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.302014 6160 scope.go:122] "RemoveContainer" containerID="b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09" Nov 02 08:42:36.311823 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.311804 6160 scope.go:122] "RemoveContainer" containerID="b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09" Nov 02 08:42:36.312087 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:42:36.312066 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09\": container with ID starting with b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09 not found: ID does not exist" containerID="b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09" Nov 02 08:42:36.312155 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.312094 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09"} err="failed to get container status \"b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09\": rpc error: code = NotFound desc = could not find container \"b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09\": container with ID starting with b14cf4b88ce8c44c1625724b80798165a7f5492fc69295e92c678e8d0eab3f09 not found: ID does not exist" Nov 02 08:42:36.443828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.443786 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb"] Nov 02 08:42:36.447635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.447569 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb"] Nov 02 08:42:36.455418 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.455388 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8e6407/distinct-w4-8e6407-kserve-7bc685c9bf-rb5qb" podUID="2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:42:36.550887 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.550839 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj"] Nov 02 08:42:36.553767 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:36.553738 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8e6407/simulator-w4-8e6407-kserve-79f85cf5f5-zg7lj"] Nov 02 08:42:37.583566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:37.583531 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" path="/var/lib/kubelet/pods/2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde/volumes" Nov 02 08:42:37.584026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:37.584011 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" path="/var/lib/kubelet/pods/5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2/volumes" Nov 02 08:42:37.894489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:37.894402 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:40.521758 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:42:40.521713 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:42:41.437847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:41.437809 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:41.931201 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:41.931157 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:42.648412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:42.648368 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:43.003716 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.003678 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n"] Nov 02 08:42:43.004274 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.004259 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" containerName="main" Nov 02 08:42:43.004334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.004274 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" containerName="main" Nov 02 08:42:43.004334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.004316 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" containerName="main" Nov 02 08:42:43.004334 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.004321 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" containerName="main" Nov 02 08:42:43.004475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.004406 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1f3909-2d7e-4382-9f2d-c9dbdeb0cfde" containerName="main" Nov 02 08:42:43.004475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.004419 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8b3a4d-3ddb-46a7-9c8d-30e3d4f39db2" containerName="main" Nov 02 08:42:43.009479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.009459 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:43.083449 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.083414 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" podUID="e3f40e31-04e8-4217-9e47-c486d3a06d80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:42:43.100814 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.100784 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9hch\" (UniqueName: \"kubernetes.io/projected/b20d3925-caaa-46ae-a508-808eb42aac2e-kube-api-access-g9hch\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n\" (UID: \"b20d3925-caaa-46ae-a508-808eb42aac2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:43.100978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.100906 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b20d3925-caaa-46ae-a508-808eb42aac2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n\" (UID: \"b20d3925-caaa-46ae-a508-808eb42aac2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:43.104768 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.104740 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9ceb40-dockercfg-thfpl\"" Nov 02 08:42:43.104768 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.104740 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9ceb40\"" Nov 02 08:42:43.108005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.107981 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n"] Nov 02 08:42:43.202693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.202651 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b20d3925-caaa-46ae-a508-808eb42aac2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n\" (UID: \"b20d3925-caaa-46ae-a508-808eb42aac2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:43.202847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.202782 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9hch\" (UniqueName: \"kubernetes.io/projected/b20d3925-caaa-46ae-a508-808eb42aac2e-kube-api-access-g9hch\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n\" (UID: \"b20d3925-caaa-46ae-a508-808eb42aac2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:43.203409 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.203390 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b20d3925-caaa-46ae-a508-808eb42aac2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n\" (UID: \"b20d3925-caaa-46ae-a508-808eb42aac2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:43.256281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.256209 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9hch\" (UniqueName: \"kubernetes.io/projected/b20d3925-caaa-46ae-a508-808eb42aac2e-kube-api-access-g9hch\") pod \"payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n\" (UID: \"b20d3925-caaa-46ae-a508-808eb42aac2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:43.322777 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.322745 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:43.695397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:43.695371 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n"] Nov 02 08:42:43.697113 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:42:43.697084 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20d3925_caaa_46ae_a508_808eb42aac2e.slice/crio-692c3ada4dbffaf13b49555339a038241e8db9b744d9edde7ae71e2903de8965 WatchSource:0}: Error finding container 692c3ada4dbffaf13b49555339a038241e8db9b744d9edde7ae71e2903de8965: Status 404 returned error can't find the container with id 692c3ada4dbffaf13b49555339a038241e8db9b744d9edde7ae71e2903de8965 Nov 02 08:42:44.373460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:44.373416 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" event={"ID":"b20d3925-caaa-46ae-a508-808eb42aac2e","Type":"ContainerStarted","Data":"c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c"} Nov 02 08:42:44.373881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:44.373476 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" event={"ID":"b20d3925-caaa-46ae-a508-808eb42aac2e","Type":"ContainerStarted","Data":"692c3ada4dbffaf13b49555339a038241e8db9b744d9edde7ae71e2903de8965"} Nov 02 08:42:44.373881 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:44.373611 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:44.480473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:44.480420 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" podStartSLOduration=2.480403247 podStartE2EDuration="2.480403247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:44.478511149 +0000 UTC m=+2163.032015346" watchObservedRunningTime="2026-11-02 08:42:44.480403247 +0000 UTC m=+2163.033907443" Nov 02 08:42:44.542691 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:44.542647 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:49.433693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:49.433647 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:52.697022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:52.696961 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:42:52.753746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:52.753719 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f40e31-04e8-4217-9e47-c486d3a06d80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f40e31-04e8-4217-9e47-c486d3a06d80-plugins-config-volume\") pod \"e3f40e31-04e8-4217-9e47-c486d3a06d80\" (UID: \"e3f40e31-04e8-4217-9e47-c486d3a06d80\") " Nov 02 08:42:52.753943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:52.753801 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f40e31-04e8-4217-9e47-c486d3a06d80-kube-api-access-blzf8\" (UniqueName: \"kubernetes.io/projected/e3f40e31-04e8-4217-9e47-c486d3a06d80-kube-api-access-blzf8\") pod \"e3f40e31-04e8-4217-9e47-c486d3a06d80\" (UID: \"e3f40e31-04e8-4217-9e47-c486d3a06d80\") " Nov 02 08:42:52.754153 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:52.754129 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f40e31-04e8-4217-9e47-c486d3a06d80-plugins-config-volume" pod "e3f40e31-04e8-4217-9e47-c486d3a06d80" (UID: "e3f40e31-04e8-4217-9e47-c486d3a06d80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:52.756055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:52.756024 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f40e31-04e8-4217-9e47-c486d3a06d80-kube-api-access-blzf8" pod "e3f40e31-04e8-4217-9e47-c486d3a06d80" (UID: "e3f40e31-04e8-4217-9e47-c486d3a06d80"). InnerVolumeSpecName "kube-api-access-blzf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:52.855162 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:52.855132 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f40e31-04e8-4217-9e47-c486d3a06d80-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:52.855162 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:52.855159 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blzf8\" (UniqueName: \"kubernetes.io/projected/e3f40e31-04e8-4217-9e47-c486d3a06d80-kube-api-access-blzf8\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:42:53.330018 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.329981 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:53.495783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.495746 6160 generic.go:356] "Generic (PLEG): container finished" podID="e3f40e31-04e8-4217-9e47-c486d3a06d80" containerID="47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1" exitCode=0 Nov 02 08:42:53.496045 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.495795 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" Nov 02 08:42:53.496158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.495812 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" event={"ID":"e3f40e31-04e8-4217-9e47-c486d3a06d80","Type":"ContainerDied","Data":"47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1"} Nov 02 08:42:53.496333 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.496299 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9" event={"ID":"e3f40e31-04e8-4217-9e47-c486d3a06d80","Type":"ContainerDied","Data":"44b4b6de75e8122a0e92cf86228072ebe9b14cce6568f2739a742c915a22ee0f"} Nov 02 08:42:53.496945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.496720 6160 scope.go:122] "RemoveContainer" containerID="47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1" Nov 02 08:42:53.511702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.511675 6160 scope.go:122] "RemoveContainer" containerID="47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1" Nov 02 08:42:53.512043 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:42:53.512021 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1\": container with ID starting with 47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1 not found: ID does not exist" containerID="47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1" Nov 02 08:42:53.512120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.512057 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1"} err="failed to get container status \"47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1\": rpc error: code = NotFound desc = could not find container \"47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1\": container with ID starting with 47482a7c02617375bb04e37a74e69ebc0402467fe8995f3299b5a55ed45a35a1 not found: ID does not exist" Nov 02 08:42:53.656439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.656351 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9"] Nov 02 08:42:53.659231 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:53.659203 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e6407-5cf6bb55f9-ndnh9"] Nov 02 08:42:54.014460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:54.014422 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:54.681571 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:54.681537 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:55.298778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:55.298738 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:55.384905 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:55.384874 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:42:55.585730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:55.585656 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f40e31-04e8-4217-9e47-c486d3a06d80" path="/var/lib/kubelet/pods/e3f40e31-04e8-4217-9e47-c486d3a06d80/volumes" Nov 02 08:42:56.789614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:56.789579 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:56.952246 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:56.952216 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:57.817222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:57.817151 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:59.537291 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:42:59.537254 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:01.766576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:01.766537 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:02.548709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:02.548665 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:03.086623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:03.086583 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:03.377383 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:03.377294 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:04.252448 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:04.252412 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:04.447997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:04.447956 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:07.014085 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:07.014043 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:07.229250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:07.229205 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:07.453299 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:07.453256 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:07.624902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:07.624863 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:08.340614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:08.340579 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:10.521166 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:43:10.521128 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:43:10.538665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:10.538635 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:10.715777 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:10.715730 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:10.925564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:10.925532 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4"] Nov 02 08:43:10.926095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:10.926083 6160 cpu_manager.go:414] "containerMap: removing container" podUID="e3f40e31-04e8-4217-9e47-c486d3a06d80" containerName="payload-processing" Nov 02 08:43:10.926140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:10.926097 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f40e31-04e8-4217-9e47-c486d3a06d80" containerName="payload-processing" Nov 02 08:43:10.926226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:10.926216 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f40e31-04e8-4217-9e47-c486d3a06d80" containerName="payload-processing" Nov 02 08:43:10.929693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:10.929677 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:10.983692 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:10.983660 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4"] Nov 02 08:43:11.033562 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.033526 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d91f6f-dockercfg-k46bc\"" Nov 02 08:43:11.033857 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.033841 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d91f6f\"" Nov 02 08:43:11.069397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.069358 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qw68\" (UniqueName: \"kubernetes.io/projected/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-kube-api-access-7qw68\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4\" (UID: \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:11.069708 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.069685 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4\" (UID: \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:11.083128 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.083099 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm"] Nov 02 08:43:11.087329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.087308 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:11.137816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.137782 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm"] Nov 02 08:43:11.171380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.171345 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qw68\" (UniqueName: \"kubernetes.io/projected/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-kube-api-access-7qw68\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4\" (UID: \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:11.171555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.171401 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4\" (UID: \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:11.172154 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.172132 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4\" (UID: \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:11.225012 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.224924 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qw68\" (UniqueName: \"kubernetes.io/projected/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-kube-api-access-7qw68\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4\" (UID: \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:11.242848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.242815 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:11.272594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.272559 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdxk\" (UniqueName: \"kubernetes.io/projected/eb26d13b-6288-4a0a-bd62-9d67e929a452-kube-api-access-6sdxk\") pod \"payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm\" (UID: \"eb26d13b-6288-4a0a-bd62-9d67e929a452\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:11.272762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.272741 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb26d13b-6288-4a0a-bd62-9d67e929a452-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm\" (UID: \"eb26d13b-6288-4a0a-bd62-9d67e929a452\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:11.374447 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.374411 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdxk\" (UniqueName: \"kubernetes.io/projected/eb26d13b-6288-4a0a-bd62-9d67e929a452-kube-api-access-6sdxk\") pod \"payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm\" (UID: \"eb26d13b-6288-4a0a-bd62-9d67e929a452\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:11.374623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.374537 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb26d13b-6288-4a0a-bd62-9d67e929a452-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm\" (UID: \"eb26d13b-6288-4a0a-bd62-9d67e929a452\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:11.375268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.375248 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb26d13b-6288-4a0a-bd62-9d67e929a452-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm\" (UID: \"eb26d13b-6288-4a0a-bd62-9d67e929a452\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:11.428051 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.428017 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdxk\" (UniqueName: \"kubernetes.io/projected/eb26d13b-6288-4a0a-bd62-9d67e929a452-kube-api-access-6sdxk\") pod \"payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm\" (UID: \"eb26d13b-6288-4a0a-bd62-9d67e929a452\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:11.612571 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.612543 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4"] Nov 02 08:43:11.614763 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:43:11.614736 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4faaadd6_625c_4ec0_a989_aed0a8a00ad6.slice/crio-ef039993bc386f41845fe93d3f11522e1e025b0b09ec2bee344f4926c7a88f60 WatchSource:0}: Error finding container ef039993bc386f41845fe93d3f11522e1e025b0b09ec2bee344f4926c7a88f60: Status 404 returned error can't find the container with id ef039993bc386f41845fe93d3f11522e1e025b0b09ec2bee344f4926c7a88f60 Nov 02 08:43:11.642315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.642278 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" event={"ID":"4faaadd6-625c-4ec0-a989-aed0a8a00ad6","Type":"ContainerStarted","Data":"ef039993bc386f41845fe93d3f11522e1e025b0b09ec2bee344f4926c7a88f60"} Nov 02 08:43:11.700405 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:11.700377 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:12.079931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:12.079900 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm"] Nov 02 08:43:12.081247 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:43:12.081218 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb26d13b_6288_4a0a_bd62_9d67e929a452.slice/crio-ef7bc768162562cc96898f948dd02d7c010404cd56158bd7a82ca973dbd8c555 WatchSource:0}: Error finding container ef7bc768162562cc96898f948dd02d7c010404cd56158bd7a82ca973dbd8c555: Status 404 returned error can't find the container with id ef7bc768162562cc96898f948dd02d7c010404cd56158bd7a82ca973dbd8c555 Nov 02 08:43:12.349753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:12.349658 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:12.651228 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:12.651103 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" event={"ID":"4faaadd6-625c-4ec0-a989-aed0a8a00ad6","Type":"ContainerStarted","Data":"8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c"} Nov 02 08:43:12.651686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:12.651302 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:12.652867 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:12.652841 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" event={"ID":"eb26d13b-6288-4a0a-bd62-9d67e929a452","Type":"ContainerStarted","Data":"54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c"} Nov 02 08:43:12.652983 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:12.652893 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" event={"ID":"eb26d13b-6288-4a0a-bd62-9d67e929a452","Type":"ContainerStarted","Data":"ef7bc768162562cc96898f948dd02d7c010404cd56158bd7a82ca973dbd8c555"} Nov 02 08:43:12.652983 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:12.652932 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:12.759464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:12.759409 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" podStartSLOduration=2.759395407 podStartE2EDuration="2.759395407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:43:12.756006564 +0000 UTC m=+2191.309510761" watchObservedRunningTime="2026-11-02 08:43:12.759395407 +0000 UTC m=+2191.312899604" Nov 02 08:43:12.861124 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:12.861075 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" podStartSLOduration=1.861059834 podStartE2EDuration="1.861059834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:43:12.860855323 +0000 UTC m=+2191.414359521" watchObservedRunningTime="2026-11-02 08:43:12.861059834 +0000 UTC m=+2191.414564032" Nov 02 08:43:17.256971 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:17.256928 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:21.268438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:21.268398 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4"] Nov 02 08:43:21.670288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:21.670249 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:21.832664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:21.832627 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:22.281921 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:22.281886 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:23.662799 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:23.662769 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:43:23.663222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:23.663045 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:43:24.953894 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:24.953856 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:26.268724 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.268681 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" podUID="69146e90-5eae-4285-83b3-146083c3e96f" containerName="payload-processing" containerID="cri-o://f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a" gracePeriod=25 Nov 02 08:43:26.529493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.529434 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:43:26.657448 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.657407 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69146e90-5eae-4285-83b3-146083c3e96f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69146e90-5eae-4285-83b3-146083c3e96f-plugins-config-volume\") pod \"69146e90-5eae-4285-83b3-146083c3e96f\" (UID: \"69146e90-5eae-4285-83b3-146083c3e96f\") " Nov 02 08:43:26.657592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.657532 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69146e90-5eae-4285-83b3-146083c3e96f-kube-api-access-jvqmx\" (UniqueName: \"kubernetes.io/projected/69146e90-5eae-4285-83b3-146083c3e96f-kube-api-access-jvqmx\") pod \"69146e90-5eae-4285-83b3-146083c3e96f\" (UID: \"69146e90-5eae-4285-83b3-146083c3e96f\") " Nov 02 08:43:26.657840 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.657818 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69146e90-5eae-4285-83b3-146083c3e96f-plugins-config-volume" pod "69146e90-5eae-4285-83b3-146083c3e96f" (UID: "69146e90-5eae-4285-83b3-146083c3e96f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:26.658048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.658032 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69146e90-5eae-4285-83b3-146083c3e96f-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:26.664538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.664513 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69146e90-5eae-4285-83b3-146083c3e96f-kube-api-access-jvqmx" pod "69146e90-5eae-4285-83b3-146083c3e96f" (UID: "69146e90-5eae-4285-83b3-146083c3e96f"). InnerVolumeSpecName "kube-api-access-jvqmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:26.759122 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.759092 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvqmx\" (UniqueName: \"kubernetes.io/projected/69146e90-5eae-4285-83b3-146083c3e96f-kube-api-access-jvqmx\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:26.775726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.775693 6160 generic.go:356] "Generic (PLEG): container finished" podID="69146e90-5eae-4285-83b3-146083c3e96f" containerID="f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a" exitCode=0 Nov 02 08:43:26.775908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.775759 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" Nov 02 08:43:26.775908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.775774 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" event={"ID":"69146e90-5eae-4285-83b3-146083c3e96f","Type":"ContainerDied","Data":"f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a"} Nov 02 08:43:26.775908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.775826 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4" event={"ID":"69146e90-5eae-4285-83b3-146083c3e96f","Type":"ContainerDied","Data":"bc4bb2026f7d3187a53d227712c702a3c9b2d326c8dedc7a5d821602443d521c"} Nov 02 08:43:26.775908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.775857 6160 scope.go:122] "RemoveContainer" containerID="f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a" Nov 02 08:43:26.783506 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.783134 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9"] Nov 02 08:43:26.783506 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.783349 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" podUID="514dee5e-6142-441c-8bba-95bab2a0be1c" containerName="maas-api" containerID="cri-o://33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19" gracePeriod=30 Nov 02 08:43:26.788787 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.788766 6160 scope.go:122] "RemoveContainer" containerID="f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a" Nov 02 08:43:26.789116 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:43:26.789091 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a\": container with ID starting with f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a not found: ID does not exist" containerID="f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a" Nov 02 08:43:26.789284 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.789124 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a"} err="failed to get container status \"f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a\": rpc error: code = NotFound desc = could not find container \"f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a\": container with ID starting with f57b79ebd36c15242b4abe1869bb14a94d6c531f894b28f028fc3d4a362fb50a not found: ID does not exist" Nov 02 08:43:26.901711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.901679 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:26.933746 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.933707 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4"] Nov 02 08:43:26.939117 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:26.939079 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4516e-56c46dd57-n8tg4"] Nov 02 08:43:27.046523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.046496 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:43:27.165642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.165613 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514dee5e-6142-441c-8bba-95bab2a0be1c-kube-api-access-qhmjh\" (UniqueName: \"kubernetes.io/projected/514dee5e-6142-441c-8bba-95bab2a0be1c-kube-api-access-qhmjh\") pod \"514dee5e-6142-441c-8bba-95bab2a0be1c\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " Nov 02 08:43:27.165825 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.165667 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514dee5e-6142-441c-8bba-95bab2a0be1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514dee5e-6142-441c-8bba-95bab2a0be1c-openshift-service-ca\") pod \"514dee5e-6142-441c-8bba-95bab2a0be1c\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " Nov 02 08:43:27.165825 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.165732 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-metrics-certs\") pod \"514dee5e-6142-441c-8bba-95bab2a0be1c\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " Nov 02 08:43:27.165825 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.165773 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls\") pod \"514dee5e-6142-441c-8bba-95bab2a0be1c\" (UID: \"514dee5e-6142-441c-8bba-95bab2a0be1c\") " Nov 02 08:43:27.166140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.166108 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514dee5e-6142-441c-8bba-95bab2a0be1c-openshift-service-ca" pod "514dee5e-6142-441c-8bba-95bab2a0be1c" (UID: "514dee5e-6142-441c-8bba-95bab2a0be1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:27.168372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.168340 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-metrics-certs" pod "514dee5e-6142-441c-8bba-95bab2a0be1c" (UID: "514dee5e-6142-441c-8bba-95bab2a0be1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:43:27.168489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.168417 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls" pod "514dee5e-6142-441c-8bba-95bab2a0be1c" (UID: "514dee5e-6142-441c-8bba-95bab2a0be1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:43:27.168564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.168495 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514dee5e-6142-441c-8bba-95bab2a0be1c-kube-api-access-qhmjh" pod "514dee5e-6142-441c-8bba-95bab2a0be1c" (UID: "514dee5e-6142-441c-8bba-95bab2a0be1c"). InnerVolumeSpecName "kube-api-access-qhmjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:27.267762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.267720 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhmjh\" (UniqueName: \"kubernetes.io/projected/514dee5e-6142-441c-8bba-95bab2a0be1c-kube-api-access-qhmjh\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:27.267762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.267755 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/514dee5e-6142-441c-8bba-95bab2a0be1c-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:27.267762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.267770 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:27.268002 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.267782 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/514dee5e-6142-441c-8bba-95bab2a0be1c-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:27.495291 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.495257 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5"] Nov 02 08:43:27.584612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.584580 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69146e90-5eae-4285-83b3-146083c3e96f" path="/var/lib/kubelet/pods/69146e90-5eae-4285-83b3-146083c3e96f/volumes" Nov 02 08:43:27.706618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.706583 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d"] Nov 02 08:43:27.776819 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.776689 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:27.785130 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.785101 6160 generic.go:356] "Generic (PLEG): container finished" podID="514dee5e-6142-441c-8bba-95bab2a0be1c" containerID="33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19" exitCode=0 Nov 02 08:43:27.785288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.785164 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" Nov 02 08:43:27.785288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.785199 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" event={"ID":"514dee5e-6142-441c-8bba-95bab2a0be1c","Type":"ContainerDied","Data":"33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19"} Nov 02 08:43:27.785288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.785262 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9" event={"ID":"514dee5e-6142-441c-8bba-95bab2a0be1c","Type":"ContainerDied","Data":"b0e1a86fb2865433c49adc2c6b4350edd71d059022a453f52e79a369ae526ca5"} Nov 02 08:43:27.785468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.785293 6160 scope.go:122] "RemoveContainer" containerID="33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19" Nov 02 08:43:27.795860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.795842 6160 scope.go:122] "RemoveContainer" containerID="33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19" Nov 02 08:43:27.796136 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:43:27.796114 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19\": container with ID starting with 33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19 not found: ID does not exist" containerID="33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19" Nov 02 08:43:27.796212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.796144 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19"} err="failed to get container status \"33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19\": rpc error: code = NotFound desc = could not find container \"33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19\": container with ID starting with 33d2dae7119ee6ff1d36fba7c6de2cc53951e5099cc5112f3fa859ff8e0fab19 not found: ID does not exist" Nov 02 08:43:27.921165 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.921105 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9"] Nov 02 08:43:27.925716 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.925687 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f4516e-d46cd85d7-wngn9"] Nov 02 08:43:27.991875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:27.991845 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:28.143595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:28.143497 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:28.725489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:28.725450 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:29.033028 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:29.032942 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:29.202257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:29.202217 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:29.584910 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:29.584880 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514dee5e-6142-441c-8bba-95bab2a0be1c" path="/var/lib/kubelet/pods/514dee5e-6142-441c-8bba-95bab2a0be1c/volumes" Nov 02 08:43:29.636266 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:29.636231 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:32.496017 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.495962 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" podUID="8a4f73f1-67e3-4516-9dd7-961ff50118e3" containerName="payload-processing" containerID="cri-o://5e6abf256f2fcd48d55e89156cf851e8fcacf6bd02d3ae755a34cc0c8bb3b2ce" gracePeriod=25 Nov 02 08:43:32.707048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.707004 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" podUID="777b2e05-d870-4293-84f2-f166f7e44667" containerName="payload-pre-processing" containerID="cri-o://12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a" gracePeriod=25 Nov 02 08:43:32.823386 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.823349 6160 generic.go:356] "Generic (PLEG): container finished" podID="8a4f73f1-67e3-4516-9dd7-961ff50118e3" containerID="5e6abf256f2fcd48d55e89156cf851e8fcacf6bd02d3ae755a34cc0c8bb3b2ce" exitCode=0 Nov 02 08:43:32.823527 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.823414 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" event={"ID":"8a4f73f1-67e3-4516-9dd7-961ff50118e3","Type":"ContainerDied","Data":"5e6abf256f2fcd48d55e89156cf851e8fcacf6bd02d3ae755a34cc0c8bb3b2ce"} Nov 02 08:43:32.823527 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.823489 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" event={"ID":"8a4f73f1-67e3-4516-9dd7-961ff50118e3","Type":"ContainerDied","Data":"28feda14af4231d508c32ba2e27cdac4759487f8a79938fd15086898f3a3b5e3"} Nov 02 08:43:32.823527 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.823507 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28feda14af4231d508c32ba2e27cdac4759487f8a79938fd15086898f3a3b5e3" Nov 02 08:43:32.841346 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.841321 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:43:32.937412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.937379 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4f73f1-67e3-4516-9dd7-961ff50118e3-kube-api-access-9j9wh\" (UniqueName: \"kubernetes.io/projected/8a4f73f1-67e3-4516-9dd7-961ff50118e3-kube-api-access-9j9wh\") pod \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\" (UID: \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\") " Nov 02 08:43:32.937614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.937501 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4f73f1-67e3-4516-9dd7-961ff50118e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4f73f1-67e3-4516-9dd7-961ff50118e3-plugins-config-volume\") pod \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\" (UID: \"8a4f73f1-67e3-4516-9dd7-961ff50118e3\") " Nov 02 08:43:32.937899 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.937872 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4f73f1-67e3-4516-9dd7-961ff50118e3-plugins-config-volume" pod "8a4f73f1-67e3-4516-9dd7-961ff50118e3" (UID: "8a4f73f1-67e3-4516-9dd7-961ff50118e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:32.940651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.940624 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4f73f1-67e3-4516-9dd7-961ff50118e3-kube-api-access-9j9wh" pod "8a4f73f1-67e3-4516-9dd7-961ff50118e3" (UID: "8a4f73f1-67e3-4516-9dd7-961ff50118e3"). InnerVolumeSpecName "kube-api-access-9j9wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:32.964188 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:32.964141 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:43:33.038392 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.038362 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777b2e05-d870-4293-84f2-f166f7e44667-kube-api-access-hdr5h\" (UniqueName: \"kubernetes.io/projected/777b2e05-d870-4293-84f2-f166f7e44667-kube-api-access-hdr5h\") pod \"777b2e05-d870-4293-84f2-f166f7e44667\" (UID: \"777b2e05-d870-4293-84f2-f166f7e44667\") " Nov 02 08:43:33.038548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.038434 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777b2e05-d870-4293-84f2-f166f7e44667-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b2e05-d870-4293-84f2-f166f7e44667-plugins-config-volume\") pod \"777b2e05-d870-4293-84f2-f166f7e44667\" (UID: \"777b2e05-d870-4293-84f2-f166f7e44667\") " Nov 02 08:43:33.038840 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.038815 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j9wh\" (UniqueName: \"kubernetes.io/projected/8a4f73f1-67e3-4516-9dd7-961ff50118e3-kube-api-access-9j9wh\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:33.038840 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.038844 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4f73f1-67e3-4516-9dd7-961ff50118e3-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:33.039011 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.038845 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777b2e05-d870-4293-84f2-f166f7e44667-plugins-config-volume" pod "777b2e05-d870-4293-84f2-f166f7e44667" (UID: "777b2e05-d870-4293-84f2-f166f7e44667"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:33.040720 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.040704 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777b2e05-d870-4293-84f2-f166f7e44667-kube-api-access-hdr5h" pod "777b2e05-d870-4293-84f2-f166f7e44667" (UID: "777b2e05-d870-4293-84f2-f166f7e44667"). InnerVolumeSpecName "kube-api-access-hdr5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:33.140397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.140308 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdr5h\" (UniqueName: \"kubernetes.io/projected/777b2e05-d870-4293-84f2-f166f7e44667-kube-api-access-hdr5h\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:33.140397 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.140340 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777b2e05-d870-4293-84f2-f166f7e44667-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:33.830831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.830748 6160 generic.go:356] "Generic (PLEG): container finished" podID="777b2e05-d870-4293-84f2-f166f7e44667" containerID="12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a" exitCode=0 Nov 02 08:43:33.830831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.830788 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" Nov 02 08:43:33.831331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.830822 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" event={"ID":"777b2e05-d870-4293-84f2-f166f7e44667","Type":"ContainerDied","Data":"12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a"} Nov 02 08:43:33.831331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.830876 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d" event={"ID":"777b2e05-d870-4293-84f2-f166f7e44667","Type":"ContainerDied","Data":"7737d6d6feb4b0cacc90acbdaa079b12df7546b2200918c220d36e8c4c58d690"} Nov 02 08:43:33.831331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.830899 6160 scope.go:122] "RemoveContainer" containerID="12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a" Nov 02 08:43:33.831331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.830959 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5" Nov 02 08:43:33.842664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.842639 6160 scope.go:122] "RemoveContainer" containerID="12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a" Nov 02 08:43:33.842938 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:43:33.842920 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a\": container with ID starting with 12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a not found: ID does not exist" containerID="12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a" Nov 02 08:43:33.842997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:33.842947 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a"} err="failed to get container status \"12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a\": rpc error: code = NotFound desc = could not find container \"12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a\": container with ID starting with 12564505e0bf5a18b902d6bf29b46bb5e2e711ece00a8cee2fda174734c4171a not found: ID does not exist" Nov 02 08:43:34.032892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:34.032851 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5"] Nov 02 08:43:34.078949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:34.078916 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d5a5a-7b76f7d7c9-prmn5"] Nov 02 08:43:34.137498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:34.137353 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d"] Nov 02 08:43:34.140527 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:34.140501 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d5a5a-76688ddcc9-kl52d"] Nov 02 08:43:35.580022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:35.579984 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:35.589579 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:35.589539 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777b2e05-d870-4293-84f2-f166f7e44667" path="/var/lib/kubelet/pods/777b2e05-d870-4293-84f2-f166f7e44667/volumes" Nov 02 08:43:35.591300 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:35.591274 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4f73f1-67e3-4516-9dd7-961ff50118e3" path="/var/lib/kubelet/pods/8a4f73f1-67e3-4516-9dd7-961ff50118e3/volumes" Nov 02 08:43:35.766280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:35.766241 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:35.942355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:35.942291 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:40.521762 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:43:40.521724 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:43:41.162962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:41.162921 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:42.314667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:42.314630 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:45.574511 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.574474 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc"] Nov 02 08:43:45.574964 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.574670 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" podUID="f530dfdb-de1d-4ea4-b590-f4f953a96cc7" containerName="maas-api" containerID="cri-o://9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50" gracePeriod=30 Nov 02 08:43:45.698152 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.698113 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:45.859068 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.859045 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:43:45.894948 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.894917 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-openshift-service-ca\") pod \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " Nov 02 08:43:45.895111 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.894974 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-metrics-certs\") pod \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " Nov 02 08:43:45.895111 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.895057 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-kube-api-access-hhd44\" (UniqueName: \"kubernetes.io/projected/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-kube-api-access-hhd44\") pod \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " Nov 02 08:43:45.895249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.895147 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls\") pod \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\" (UID: \"f530dfdb-de1d-4ea4-b590-f4f953a96cc7\") " Nov 02 08:43:45.895386 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.895359 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-openshift-service-ca" pod "f530dfdb-de1d-4ea4-b590-f4f953a96cc7" (UID: "f530dfdb-de1d-4ea4-b590-f4f953a96cc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:45.895567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.895548 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:45.897405 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.897375 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls" pod "f530dfdb-de1d-4ea4-b590-f4f953a96cc7" (UID: "f530dfdb-de1d-4ea4-b590-f4f953a96cc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:43:45.897514 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.897401 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-metrics-certs" pod "f530dfdb-de1d-4ea4-b590-f4f953a96cc7" (UID: "f530dfdb-de1d-4ea4-b590-f4f953a96cc7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:43:45.897514 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.897452 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-kube-api-access-hhd44" pod "f530dfdb-de1d-4ea4-b590-f4f953a96cc7" (UID: "f530dfdb-de1d-4ea4-b590-f4f953a96cc7"). InnerVolumeSpecName "kube-api-access-hhd44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:45.915974 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.915948 6160 generic.go:356] "Generic (PLEG): container finished" podID="f530dfdb-de1d-4ea4-b590-f4f953a96cc7" containerID="9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50" exitCode=0 Nov 02 08:43:45.916099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.915992 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" Nov 02 08:43:45.916099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.916024 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" event={"ID":"f530dfdb-de1d-4ea4-b590-f4f953a96cc7","Type":"ContainerDied","Data":"9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50"} Nov 02 08:43:45.916099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.916081 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc" event={"ID":"f530dfdb-de1d-4ea4-b590-f4f953a96cc7","Type":"ContainerDied","Data":"fff0e4dd08357fa97099404e656777d099005c4db993b53c6d3f864b040df7bb"} Nov 02 08:43:45.916306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.916110 6160 scope.go:122] "RemoveContainer" containerID="9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50" Nov 02 08:43:45.926460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.926444 6160 scope.go:122] "RemoveContainer" containerID="9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50" Nov 02 08:43:45.926743 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:43:45.926714 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50\": container with ID starting with 9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50 not found: ID does not exist" containerID="9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50" Nov 02 08:43:45.926842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.926748 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50"} err="failed to get container status \"9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50\": rpc error: code = NotFound desc = could not find container \"9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50\": container with ID starting with 9d427783e747506e68abc112acf0ddbafe017ff252d3bec4729bda6814661c50 not found: ID does not exist" Nov 02 08:43:46.005219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.997708 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhd44\" (UniqueName: \"kubernetes.io/projected/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-kube-api-access-hhd44\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:46.005219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.997781 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:46.005219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:45.997802 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f530dfdb-de1d-4ea4-b590-f4f953a96cc7-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:46.071533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:46.071495 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc"] Nov 02 08:43:46.078501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:46.078466 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e6407-56bbb56cd9-x7prc"] Nov 02 08:43:47.404308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:47.404256 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:47.585595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:47.585559 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f530dfdb-de1d-4ea4-b590-f4f953a96cc7" path="/var/lib/kubelet/pods/f530dfdb-de1d-4ea4-b590-f4f953a96cc7/volumes" Nov 02 08:43:47.642702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:47.642668 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:47.860202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:47.860146 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:48.123723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:48.123634 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv"] Nov 02 08:43:48.232633 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:48.232578 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd"] Nov 02 08:43:48.530342 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:48.530312 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:48.706083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:48.706036 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:50.466457 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:50.466412 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:53.124444 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.124374 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" podUID="a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab" containerName="payload-processing" containerID="cri-o://b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948" gracePeriod=25 Nov 02 08:43:53.234289 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.234241 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" podUID="8a6b737b-8aeb-47e7-920a-33a1d396b983" containerName="payload-pre-processing" containerID="cri-o://0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc" gracePeriod=25 Nov 02 08:43:53.410608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.410581 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:43:53.484732 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.484699 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-kube-api-access-mgtb5\" (UniqueName: \"kubernetes.io/projected/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-kube-api-access-mgtb5\") pod \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\" (UID: \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\") " Nov 02 08:43:53.484903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.484805 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-plugins-config-volume\") pod \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\" (UID: \"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab\") " Nov 02 08:43:53.485199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.485144 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-plugins-config-volume" pod "a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab" (UID: "a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:53.485308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.485289 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:53.486995 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.486970 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-kube-api-access-mgtb5" pod "a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab" (UID: "a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab"). InnerVolumeSpecName "kube-api-access-mgtb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:53.497655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.497630 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:43:53.586200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.586152 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6b737b-8aeb-47e7-920a-33a1d396b983-kube-api-access-vsmqd\" (UniqueName: \"kubernetes.io/projected/8a6b737b-8aeb-47e7-920a-33a1d396b983-kube-api-access-vsmqd\") pod \"8a6b737b-8aeb-47e7-920a-33a1d396b983\" (UID: \"8a6b737b-8aeb-47e7-920a-33a1d396b983\") " Nov 02 08:43:53.586396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.586287 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a6b737b-8aeb-47e7-920a-33a1d396b983-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b737b-8aeb-47e7-920a-33a1d396b983-plugins-config-volume\") pod \"8a6b737b-8aeb-47e7-920a-33a1d396b983\" (UID: \"8a6b737b-8aeb-47e7-920a-33a1d396b983\") " Nov 02 08:43:53.586589 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.586565 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgtb5\" (UniqueName: \"kubernetes.io/projected/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab-kube-api-access-mgtb5\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:53.586693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.586649 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6b737b-8aeb-47e7-920a-33a1d396b983-plugins-config-volume" pod "8a6b737b-8aeb-47e7-920a-33a1d396b983" (UID: "8a6b737b-8aeb-47e7-920a-33a1d396b983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:43:53.588436 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.588416 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6b737b-8aeb-47e7-920a-33a1d396b983-kube-api-access-vsmqd" pod "8a6b737b-8aeb-47e7-920a-33a1d396b983" (UID: "8a6b737b-8aeb-47e7-920a-33a1d396b983"). InnerVolumeSpecName "kube-api-access-vsmqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:43:53.688034 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.687996 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsmqd\" (UniqueName: \"kubernetes.io/projected/8a6b737b-8aeb-47e7-920a-33a1d396b983-kube-api-access-vsmqd\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:53.688034 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.688025 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b737b-8aeb-47e7-920a-33a1d396b983-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:43:53.979509 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.979410 6160 generic.go:356] "Generic (PLEG): container finished" podID="8a6b737b-8aeb-47e7-920a-33a1d396b983" containerID="0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc" exitCode=0 Nov 02 08:43:53.979509 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.979461 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" Nov 02 08:43:53.979730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.979502 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" event={"ID":"8a6b737b-8aeb-47e7-920a-33a1d396b983","Type":"ContainerDied","Data":"0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc"} Nov 02 08:43:53.979730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.979566 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd" event={"ID":"8a6b737b-8aeb-47e7-920a-33a1d396b983","Type":"ContainerDied","Data":"123e0de7103f2a6df6350ca431ee913188d25b66e7bd98bd2fe51a7c48865f78"} Nov 02 08:43:53.979730 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.979589 6160 scope.go:122] "RemoveContainer" containerID="0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc" Nov 02 08:43:53.981274 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.981248 6160 generic.go:356] "Generic (PLEG): container finished" podID="a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab" containerID="b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948" exitCode=0 Nov 02 08:43:53.981385 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.981289 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" Nov 02 08:43:53.981385 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.981305 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" event={"ID":"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab","Type":"ContainerDied","Data":"b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948"} Nov 02 08:43:53.981385 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.981362 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv" event={"ID":"a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab","Type":"ContainerDied","Data":"5109838f2950f753f323fbad09cdedf38d748c12c84251c92ed949bf25be17bc"} Nov 02 08:43:53.991103 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.991083 6160 scope.go:122] "RemoveContainer" containerID="0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc" Nov 02 08:43:53.991420 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:43:53.991401 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc\": container with ID starting with 0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc not found: ID does not exist" containerID="0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc" Nov 02 08:43:53.991499 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.991427 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc"} err="failed to get container status \"0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc\": rpc error: code = NotFound desc = could not find container \"0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc\": container with ID starting with 0e93e73168b6091bfc73550a9364080c5b3136d69b98ec26ce14d8f414ae32dc not found: ID does not exist" Nov 02 08:43:53.991499 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:53.991444 6160 scope.go:122] "RemoveContainer" containerID="b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948" Nov 02 08:43:54.000689 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.000670 6160 scope.go:122] "RemoveContainer" containerID="b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948" Nov 02 08:43:54.000943 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:43:54.000924 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948\": container with ID starting with b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948 not found: ID does not exist" containerID="b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948" Nov 02 08:43:54.000991 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.000950 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948"} err="failed to get container status \"b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948\": rpc error: code = NotFound desc = could not find container \"b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948\": container with ID starting with b667e8672e12539513650b432ce2d7be4fecd7190f12a6f59d463d2ef4a13948 not found: ID does not exist" Nov 02 08:43:54.135182 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.135151 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd"] Nov 02 08:43:54.181481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.181449 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad4580-7c4f7599f9-zf8kd"] Nov 02 08:43:54.276240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.276144 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh"] Nov 02 08:43:54.282611 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.282585 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv"] Nov 02 08:43:54.285157 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.285135 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad4580-c6756f6f7-dqstv"] Nov 02 08:43:54.566518 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.566434 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:54.770167 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.770131 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:54.996757 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:54.996715 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:55.584751 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:55.584714 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a6b737b-8aeb-47e7-920a-33a1d396b983" path="/var/lib/kubelet/pods/8a6b737b-8aeb-47e7-920a-33a1d396b983/volumes" Nov 02 08:43:55.585148 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:55.585117 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab" path="/var/lib/kubelet/pods/a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab/volumes" Nov 02 08:43:58.430299 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:58.430258 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:59.277196 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.277113 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" podUID="466de682-78b8-4b8f-8e6e-6fd1205f41b2" containerName="payload-pre-processing" containerID="cri-o://b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84" gracePeriod=25 Nov 02 08:43:59.861735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.861700 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b"] Nov 02 08:43:59.862352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862336 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab" containerName="payload-processing" Nov 02 08:43:59.862405 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862354 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab" containerName="payload-processing" Nov 02 08:43:59.862405 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862374 6160 cpu_manager.go:414] "containerMap: removing container" podUID="69146e90-5eae-4285-83b3-146083c3e96f" containerName="payload-processing" Nov 02 08:43:59.862405 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862379 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69146e90-5eae-4285-83b3-146083c3e96f" containerName="payload-processing" Nov 02 08:43:59.862405 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862400 6160 cpu_manager.go:414] "containerMap: removing container" podUID="8a4f73f1-67e3-4516-9dd7-961ff50118e3" containerName="payload-processing" Nov 02 08:43:59.862532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862406 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4f73f1-67e3-4516-9dd7-961ff50118e3" containerName="payload-processing" Nov 02 08:43:59.862532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862416 6160 cpu_manager.go:414] "containerMap: removing container" podUID="8a6b737b-8aeb-47e7-920a-33a1d396b983" containerName="payload-pre-processing" Nov 02 08:43:59.862532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862421 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6b737b-8aeb-47e7-920a-33a1d396b983" containerName="payload-pre-processing" Nov 02 08:43:59.862532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862428 6160 cpu_manager.go:414] "containerMap: removing container" podUID="777b2e05-d870-4293-84f2-f166f7e44667" containerName="payload-pre-processing" Nov 02 08:43:59.862532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862433 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b2e05-d870-4293-84f2-f166f7e44667" containerName="payload-pre-processing" Nov 02 08:43:59.862532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862445 6160 cpu_manager.go:414] "containerMap: removing container" podUID="f530dfdb-de1d-4ea4-b590-f4f953a96cc7" containerName="maas-api" Nov 02 08:43:59.862532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862450 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f530dfdb-de1d-4ea4-b590-f4f953a96cc7" containerName="maas-api" Nov 02 08:43:59.862532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862461 6160 cpu_manager.go:414] "containerMap: removing container" podUID="514dee5e-6142-441c-8bba-95bab2a0be1c" containerName="maas-api" Nov 02 08:43:59.862532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862465 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514dee5e-6142-441c-8bba-95bab2a0be1c" containerName="maas-api" Nov 02 08:43:59.862797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862543 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4f73f1-67e3-4516-9dd7-961ff50118e3" containerName="payload-processing" Nov 02 08:43:59.862797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862553 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6b737b-8aeb-47e7-920a-33a1d396b983" containerName="payload-pre-processing" Nov 02 08:43:59.862797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862558 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="777b2e05-d870-4293-84f2-f166f7e44667" containerName="payload-pre-processing" Nov 02 08:43:59.862797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862565 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="f530dfdb-de1d-4ea4-b590-f4f953a96cc7" containerName="maas-api" Nov 02 08:43:59.862797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862572 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="514dee5e-6142-441c-8bba-95bab2a0be1c" containerName="maas-api" Nov 02 08:43:59.862797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862580 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2bd9e6f-8151-4dd1-af9b-b9f500d4a0ab" containerName="payload-processing" Nov 02 08:43:59.862797 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.862586 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="69146e90-5eae-4285-83b3-146083c3e96f" containerName="payload-processing" Nov 02 08:43:59.867553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.867535 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:43:59.917640 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.917606 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d91f6f-serving-cert\"" Nov 02 08:43:59.976519 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.976271 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:43:59.976747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.976572 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-metrics-certs\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:43:59.976816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.976738 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnql6\" (UniqueName: \"kubernetes.io/projected/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-kube-api-access-dnql6\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:43:59.976961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.976927 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:43:59.979407 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:43:59.979381 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b"] Nov 02 08:44:00.078388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.078349 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnql6\" (UniqueName: \"kubernetes.io/projected/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-kube-api-access-dnql6\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:00.078584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.078460 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:00.078584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.078493 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:00.078584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.078526 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-metrics-certs\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:00.078790 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:44:00.078603 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-serving-cert: secret "maas-api-e2e-ipp-b-d91f6f-serving-cert" not found Nov 02 08:44:00.078790 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:44:00.078707 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls podName:c821cfa2-aa32-4fa8-a9b6-f52884cd658a nodeName:}" failed. No retries permitted until 2026-11-02 08:44:00.578683542 +0000 UTC m=+2239.132187730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls") pod "maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" (UID: "c821cfa2-aa32-4fa8-a9b6-f52884cd658a") : secret "maas-api-e2e-ipp-b-d91f6f-serving-cert" not found Nov 02 08:44:00.079293 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.079263 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:00.081376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.081353 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-metrics-certs\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:00.132626 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.132552 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnql6\" (UniqueName: \"kubernetes.io/projected/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-kube-api-access-dnql6\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:00.139693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.139664 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:00.251669 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.251627 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" podUID="466de682-78b8-4b8f-8e6e-6fd1205f41b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:44:00.293412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.293376 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:00.451868 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.451830 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:00.583877 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.583846 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:00.586483 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.586455 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:00.782529 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:00.782393 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:01.155294 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:01.155166 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b"] Nov 02 08:44:01.157085 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:44:01.157053 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc821cfa2_aa32_4fa8_a9b6_f52884cd658a.slice/crio-f8d1630c46cf0ff68f4ed581d127cd6550df7021d7675747fb3a9164d7faee57 WatchSource:0}: Error finding container f8d1630c46cf0ff68f4ed581d127cd6550df7021d7675747fb3a9164d7faee57: Status 404 returned error can't find the container with id f8d1630c46cf0ff68f4ed581d127cd6550df7021d7675747fb3a9164d7faee57 Nov 02 08:44:02.040774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:02.040738 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" event={"ID":"c821cfa2-aa32-4fa8-a9b6-f52884cd658a","Type":"ContainerStarted","Data":"f8d1630c46cf0ff68f4ed581d127cd6550df7021d7675747fb3a9164d7faee57"} Nov 02 08:44:03.048829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:03.048793 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" event={"ID":"c821cfa2-aa32-4fa8-a9b6-f52884cd658a","Type":"ContainerStarted","Data":"01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24"} Nov 02 08:44:03.049263 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:03.048883 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:03.157642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:03.157585 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" podStartSLOduration=3.134838578 podStartE2EDuration="4.157570012s" totalImagesPullingTime="1.022731434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:44:03.15523653 +0000 UTC m=+2241.708740746" watchObservedRunningTime="2026-11-02 08:44:03.157570012 +0000 UTC m=+2241.711074208" Nov 02 08:44:04.224384 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:04.224353 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:04.401099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:04.401059 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:04.580007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:04.579921 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:05.540982 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:05.540944 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:08.671623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:08.671587 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:09.060970 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:09.060885 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:44:10.251374 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:10.251337 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" podUID="466de682-78b8-4b8f-8e6e-6fd1205f41b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:44:10.342473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:10.342422 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:10.521998 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:44:10.521914 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:44:15.312521 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:15.312486 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:18.835440 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:18.835397 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:19.329168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:19.329131 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:19.559473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:19.559448 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:44:19.605727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:19.605687 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466de682-78b8-4b8f-8e6e-6fd1205f41b2-kube-api-access-pxhxd\" (UniqueName: \"kubernetes.io/projected/466de682-78b8-4b8f-8e6e-6fd1205f41b2-kube-api-access-pxhxd\") pod \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\" (UID: \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\") " Nov 02 08:44:19.605727 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:19.605747 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466de682-78b8-4b8f-8e6e-6fd1205f41b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466de682-78b8-4b8f-8e6e-6fd1205f41b2-plugins-config-volume\") pod \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\" (UID: \"466de682-78b8-4b8f-8e6e-6fd1205f41b2\") " Nov 02 08:44:19.606377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:19.606121 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466de682-78b8-4b8f-8e6e-6fd1205f41b2-plugins-config-volume" pod "466de682-78b8-4b8f-8e6e-6fd1205f41b2" (UID: "466de682-78b8-4b8f-8e6e-6fd1205f41b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:19.606377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:19.606232 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/466de682-78b8-4b8f-8e6e-6fd1205f41b2-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:19.608119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:19.608097 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466de682-78b8-4b8f-8e6e-6fd1205f41b2-kube-api-access-pxhxd" pod "466de682-78b8-4b8f-8e6e-6fd1205f41b2" (UID: "466de682-78b8-4b8f-8e6e-6fd1205f41b2"). InnerVolumeSpecName "kube-api-access-pxhxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:19.707355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:19.707309 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxhxd\" (UniqueName: \"kubernetes.io/projected/466de682-78b8-4b8f-8e6e-6fd1205f41b2-kube-api-access-pxhxd\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:20.172370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:20.172286 6160 generic.go:356] "Generic (PLEG): container finished" podID="466de682-78b8-4b8f-8e6e-6fd1205f41b2" containerID="b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84" exitCode=0 Nov 02 08:44:20.172370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:20.172329 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" Nov 02 08:44:20.172893 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:20.172383 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" event={"ID":"466de682-78b8-4b8f-8e6e-6fd1205f41b2","Type":"ContainerDied","Data":"b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84"} Nov 02 08:44:20.172893 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:20.172445 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh" event={"ID":"466de682-78b8-4b8f-8e6e-6fd1205f41b2","Type":"ContainerDied","Data":"3a739ccce868266bc5919da5aedc1f81578ed1925800675631fb778e32e0fdcb"} Nov 02 08:44:20.172893 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:20.172479 6160 scope.go:122] "RemoveContainer" containerID="b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84" Nov 02 08:44:20.182704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:20.182688 6160 scope.go:122] "RemoveContainer" containerID="b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84" Nov 02 08:44:20.182955 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:44:20.182939 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84\": container with ID starting with b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84 not found: ID does not exist" containerID="b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84" Nov 02 08:44:20.183010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:20.182964 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84"} err="failed to get container status \"b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84\": rpc error: code = NotFound desc = could not find container \"b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84\": container with ID starting with b79f666f25b428b53aaa229ca2e4b1fa86d9d799d0f5a50659de1b0a1432bf84 not found: ID does not exist" Nov 02 08:44:20.325841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:20.325801 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh"] Nov 02 08:44:20.372031 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:20.371982 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e6ca80-d8dc59779-b5trh"] Nov 02 08:44:21.084075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:21.084042 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw"] Nov 02 08:44:21.084282 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:21.084262 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" containerID="cri-o://f36b75aa29b6f7340bf475ea6d332c084c39a4d40e481d089c4c3eaf52aeb39d" gracePeriod=30 Nov 02 08:44:21.585139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:21.585107 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466de682-78b8-4b8f-8e6e-6fd1205f41b2" path="/var/lib/kubelet/pods/466de682-78b8-4b8f-8e6e-6fd1205f41b2/volumes" Nov 02 08:44:22.048591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:22.048554 6160 patch_prober.go:29] interesting pod/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:44:22.048766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:22.048612 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:44:22.344529 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:22.344442 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:25.283918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:25.283873 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:25.610956 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:25.610844 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:25.767315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:25.767276 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:25.952076 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:25.952021 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:26.213163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.213122 6160 generic.go:356] "Generic (PLEG): container finished" podID="946115c9-5d90-431d-8213-9724c9f1a430" containerID="f36b75aa29b6f7340bf475ea6d332c084c39a4d40e481d089c4c3eaf52aeb39d" exitCode=0 Nov 02 08:44:26.213358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.213205 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" event={"ID":"946115c9-5d90-431d-8213-9724c9f1a430","Type":"ContainerDied","Data":"f36b75aa29b6f7340bf475ea6d332c084c39a4d40e481d089c4c3eaf52aeb39d"} Nov 02 08:44:26.347666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.347638 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:44:26.484552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484472 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-socket\") pod \"946115c9-5d90-431d-8213-9724c9f1a430\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " Nov 02 08:44:26.484552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484510 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-istio-token\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-istio-token\") pod \"946115c9-5d90-431d-8213-9724c9f1a430\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " Nov 02 08:44:26.484552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484542 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-certs\") pod \"946115c9-5d90-431d-8213-9724c9f1a430\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " Nov 02 08:44:26.484852 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484687 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-credential-socket\") pod \"946115c9-5d90-431d-8213-9724c9f1a430\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " Nov 02 08:44:26.484852 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484735 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-kube-api-access-tjxwm\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-kube-api-access-tjxwm\") pod \"946115c9-5d90-431d-8213-9724c9f1a430\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " Nov 02 08:44:26.484852 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484792 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-socket" pod "946115c9-5d90-431d-8213-9724c9f1a430" (UID: "946115c9-5d90-431d-8213-9724c9f1a430"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:26.484852 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484831 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/946115c9-5d90-431d-8213-9724c9f1a430-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946115c9-5d90-431d-8213-9724c9f1a430-istio-podinfo\") pod \"946115c9-5d90-431d-8213-9724c9f1a430\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " Nov 02 08:44:26.484852 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484840 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-certs" pod "946115c9-5d90-431d-8213-9724c9f1a430" (UID: "946115c9-5d90-431d-8213-9724c9f1a430"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:26.485118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484897 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-envoy\") pod \"946115c9-5d90-431d-8213-9724c9f1a430\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " Nov 02 08:44:26.485118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484909 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-credential-socket" pod "946115c9-5d90-431d-8213-9724c9f1a430" (UID: "946115c9-5d90-431d-8213-9724c9f1a430"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:26.485118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.484956 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946115c9-5d90-431d-8213-9724c9f1a430-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946115c9-5d90-431d-8213-9724c9f1a430-istiod-ca-cert\") pod \"946115c9-5d90-431d-8213-9724c9f1a430\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " Nov 02 08:44:26.485118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.485017 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-data\") pod \"946115c9-5d90-431d-8213-9724c9f1a430\" (UID: \"946115c9-5d90-431d-8213-9724c9f1a430\") " Nov 02 08:44:26.485377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.485349 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-data" pod "946115c9-5d90-431d-8213-9724c9f1a430" (UID: "946115c9-5d90-431d-8213-9724c9f1a430"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:26.485594 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.485570 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946115c9-5d90-431d-8213-9724c9f1a430-istiod-ca-cert" pod "946115c9-5d90-431d-8213-9724c9f1a430" (UID: "946115c9-5d90-431d-8213-9724c9f1a430"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:26.485688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.485634 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:26.485688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.485656 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:26.485688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.485670 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:26.485814 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.485686 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:26.486991 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.486956 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-istio-token" pod "946115c9-5d90-431d-8213-9724c9f1a430" (UID: "946115c9-5d90-431d-8213-9724c9f1a430"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:26.487159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.487140 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-envoy" pod "946115c9-5d90-431d-8213-9724c9f1a430" (UID: "946115c9-5d90-431d-8213-9724c9f1a430"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:26.487560 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.487543 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/946115c9-5d90-431d-8213-9724c9f1a430-istio-podinfo" pod "946115c9-5d90-431d-8213-9724c9f1a430" (UID: "946115c9-5d90-431d-8213-9724c9f1a430"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:44:26.487628 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.487547 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-kube-api-access-tjxwm" pod "946115c9-5d90-431d-8213-9724c9f1a430" (UID: "946115c9-5d90-431d-8213-9724c9f1a430"). InnerVolumeSpecName "kube-api-access-tjxwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:26.587008 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.586975 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946115c9-5d90-431d-8213-9724c9f1a430-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:26.587008 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.587003 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:26.587008 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.587013 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxwm\" (UniqueName: \"kubernetes.io/projected/946115c9-5d90-431d-8213-9724c9f1a430-kube-api-access-tjxwm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:26.587280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.587022 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946115c9-5d90-431d-8213-9724c9f1a430-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:26.587280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:26.587032 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946115c9-5d90-431d-8213-9724c9f1a430-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:44:27.221022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:27.220991 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" Nov 02 08:44:27.221022 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:27.221007 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw" event={"ID":"946115c9-5d90-431d-8213-9724c9f1a430","Type":"ContainerDied","Data":"d3e93b6d98c9fa89eec57adb4a7a421bbbe4c93830f25bb46533985a9dc3d0b9"} Nov 02 08:44:27.221333 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:27.221065 6160 scope.go:122] "RemoveContainer" containerID="f36b75aa29b6f7340bf475ea6d332c084c39a4d40e481d089c4c3eaf52aeb39d" Nov 02 08:44:27.375548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:27.375503 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw"] Nov 02 08:44:27.384320 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:27.384290 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d0628e-openshift-default-6c484c4c6f-qvwxw"] Nov 02 08:44:27.584765 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:27.584689 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946115c9-5d90-431d-8213-9724c9f1a430" path="/var/lib/kubelet/pods/946115c9-5d90-431d-8213-9724c9f1a430/volumes" Nov 02 08:44:28.847728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:28.847694 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:31.749593 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:31.749529 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:31.914335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:31.914296 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:33.039962 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:33.039921 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:34.573610 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:34.573577 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:34.844969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:34.844877 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:35.007114 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:35.007074 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:35.501932 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:35.501898 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:40.521348 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:44:40.521314 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:44:41.374290 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:41.374260 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:41.547060 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:41.547020 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:41.723609 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:41.723569 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:41.886304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:41.886260 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:42.108453 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:42.108361 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:47.030106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:47.030072 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt"] Nov 02 08:44:47.035329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:47.035303 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8"] Nov 02 08:44:47.150417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:47.150373 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:47.364672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:47.364566 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:52.575442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:52.574968 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:54.119158 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:54.119116 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8"] Nov 02 08:44:54.229052 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:54.229016 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx"] Nov 02 08:44:54.513399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:54.513356 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:54.676616 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:54.676576 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:58.109774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:58.109726 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:59.119908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:59.119848 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" podUID="0434f554-d0b4-4dfb-9d87-efa7039b344c" containerName="payload-processing" containerID="cri-o://21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344" gracePeriod=25 Nov 02 08:44:59.230358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:44:59.230305 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" podUID="202b4ba4-e975-4a6d-addc-37a40849ef7f" containerName="payload-pre-processing" containerID="cri-o://6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571" gracePeriod=25 Nov 02 08:45:00.189105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.189068 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7"] Nov 02 08:45:00.189653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.189635 6160 cpu_manager.go:414] "containerMap: removing container" podUID="466de682-78b8-4b8f-8e6e-6fd1205f41b2" containerName="payload-pre-processing" Nov 02 08:45:00.189703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.189654 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466de682-78b8-4b8f-8e6e-6fd1205f41b2" containerName="payload-pre-processing" Nov 02 08:45:00.189703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.189682 6160 cpu_manager.go:414] "containerMap: removing container" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" Nov 02 08:45:00.189703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.189687 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" Nov 02 08:45:00.189804 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.189773 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="466de682-78b8-4b8f-8e6e-6fd1205f41b2" containerName="payload-pre-processing" Nov 02 08:45:00.189804 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.189785 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="946115c9-5d90-431d-8213-9724c9f1a430" containerName="istio-proxy" Nov 02 08:45:00.195116 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.195099 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:00.243435 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.243400 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:45:00.287406 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.287363 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68726ce2-0477-4746-a217-cc5bb9548622-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7\" (UID: \"68726ce2-0477-4746-a217-cc5bb9548622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:00.287630 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.287610 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqfrv\" (UniqueName: \"kubernetes.io/projected/68726ce2-0477-4746-a217-cc5bb9548622-kube-api-access-bqfrv\") pod \"maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7\" (UID: \"68726ce2-0477-4746-a217-cc5bb9548622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:00.376415 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.376379 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7"] Nov 02 08:45:00.389309 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.389277 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqfrv\" (UniqueName: \"kubernetes.io/projected/68726ce2-0477-4746-a217-cc5bb9548622-kube-api-access-bqfrv\") pod \"maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7\" (UID: \"68726ce2-0477-4746-a217-cc5bb9548622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:00.389510 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.389385 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68726ce2-0477-4746-a217-cc5bb9548622-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7\" (UID: \"68726ce2-0477-4746-a217-cc5bb9548622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:00.390066 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.390043 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68726ce2-0477-4746-a217-cc5bb9548622-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7\" (UID: \"68726ce2-0477-4746-a217-cc5bb9548622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:00.442612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.442539 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqfrv\" (UniqueName: \"kubernetes.io/projected/68726ce2-0477-4746-a217-cc5bb9548622-kube-api-access-bqfrv\") pod \"maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7\" (UID: \"68726ce2-0477-4746-a217-cc5bb9548622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:00.510010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.509976 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:00.879900 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:00.879877 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7"] Nov 02 08:45:00.882140 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:45:00.882109 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68726ce2_0477_4746_a217_cc5bb9548622.slice/crio-fbccbc43c2910e14af5168dcf1d73d6354ae5ef6e4b4b2ed426997006c7bbce9 WatchSource:0}: Error finding container fbccbc43c2910e14af5168dcf1d73d6354ae5ef6e4b4b2ed426997006c7bbce9: Status 404 returned error can't find the container with id fbccbc43c2910e14af5168dcf1d73d6354ae5ef6e4b4b2ed426997006c7bbce9 Nov 02 08:45:01.455465 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:01.455421 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" event={"ID":"68726ce2-0477-4746-a217-cc5bb9548622","Type":"ContainerStarted","Data":"9895f82448d30deac2c42f8625d46348d6088aa4ae117edf2dc989cd29c1683d"} Nov 02 08:45:01.455907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:01.455484 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" event={"ID":"68726ce2-0477-4746-a217-cc5bb9548622","Type":"ContainerStarted","Data":"fbccbc43c2910e14af5168dcf1d73d6354ae5ef6e4b4b2ed426997006c7bbce9"} Nov 02 08:45:01.562563 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:01.562457 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" podStartSLOduration=1.562444407 podStartE2EDuration="1.562444407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:01.561649226 +0000 UTC m=+2300.115153447" watchObservedRunningTime="2026-11-02 08:45:01.562444407 +0000 UTC m=+2300.115948605" Nov 02 08:45:02.054570 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.054534 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" podUID="a0176aba-5186-4993-84f9-7e5c8b79d1a9" containerName="main" containerID="cri-o://cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39" gracePeriod=45 Nov 02 08:45:02.058677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.058631 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" podUID="af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" containerName="main" containerID="cri-o://e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa" gracePeriod=45 Nov 02 08:45:02.351159 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.351133 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:45:02.354299 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.354277 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:45:02.464977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.464927 6160 generic.go:356] "Generic (PLEG): container finished" podID="a0176aba-5186-4993-84f9-7e5c8b79d1a9" containerID="cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39" exitCode=0 Nov 02 08:45:02.464977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.464972 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" Nov 02 08:45:02.465567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.465057 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" event={"ID":"a0176aba-5186-4993-84f9-7e5c8b79d1a9","Type":"ContainerDied","Data":"cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39"} Nov 02 08:45:02.465567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.465091 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt" event={"ID":"a0176aba-5186-4993-84f9-7e5c8b79d1a9","Type":"ContainerDied","Data":"e0a9db2a37d7fe8bf26c86d2b21050dfe0a045fa55b4bf02a8d6700d34c54411"} Nov 02 08:45:02.465567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.465112 6160 scope.go:122] "RemoveContainer" containerID="cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39" Nov 02 08:45:02.468551 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.467344 6160 generic.go:356] "Generic (PLEG): container finished" podID="68726ce2-0477-4746-a217-cc5bb9548622" containerID="9895f82448d30deac2c42f8625d46348d6088aa4ae117edf2dc989cd29c1683d" exitCode=60 Nov 02 08:45:02.468551 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.467410 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" event={"ID":"68726ce2-0477-4746-a217-cc5bb9548622","Type":"ContainerDied","Data":"9895f82448d30deac2c42f8625d46348d6088aa4ae117edf2dc989cd29c1683d"} Nov 02 08:45:02.468551 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.468037 6160 scope.go:122] "RemoveContainer" containerID="9895f82448d30deac2c42f8625d46348d6088aa4ae117edf2dc989cd29c1683d" Nov 02 08:45:02.472706 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.472005 6160 generic.go:356] "Generic (PLEG): container finished" podID="af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" containerID="e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa" exitCode=0 Nov 02 08:45:02.472706 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.472082 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" event={"ID":"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2","Type":"ContainerDied","Data":"e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa"} Nov 02 08:45:02.472706 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.472116 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" event={"ID":"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2","Type":"ContainerDied","Data":"31adea840182ba4c01f34a802c3b6a706e36baccf35a4ed1762fe54e2a599472"} Nov 02 08:45:02.472706 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.472205 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" Nov 02 08:45:02.481572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.481337 6160 scope.go:122] "RemoveContainer" containerID="cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39" Nov 02 08:45:02.481711 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:02.481642 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39\": container with ID starting with cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39 not found: ID does not exist" containerID="cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39" Nov 02 08:45:02.481711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.481678 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39"} err="failed to get container status \"cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39\": rpc error: code = NotFound desc = could not find container \"cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39\": container with ID starting with cbb3d98ce080635c14ebd7721394f66e0bed5a3fedee08bb09a024a40a135e39 not found: ID does not exist" Nov 02 08:45:02.481711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.481703 6160 scope.go:122] "RemoveContainer" containerID="e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa" Nov 02 08:45:02.492388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.492277 6160 scope.go:122] "RemoveContainer" containerID="e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa" Nov 02 08:45:02.492673 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:02.492630 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa\": container with ID starting with e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa not found: ID does not exist" containerID="e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa" Nov 02 08:45:02.492769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.492667 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa"} err="failed to get container status \"e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa\": rpc error: code = NotFound desc = could not find container \"e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa\": container with ID starting with e70e7f78163886ea50864131b25d9591d9ebfb2103b5c9284fd07eb00e0a62aa not found: ID does not exist" Nov 02 08:45:02.514240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514211 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-home\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-home\") pod \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " Nov 02 08:45:02.514344 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514307 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-model-cache\") pod \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " Nov 02 08:45:02.514384 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514356 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-kube-api-access-c8xdl\" (UniqueName: \"kubernetes.io/projected/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-kube-api-access-c8xdl\") pod \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " Nov 02 08:45:02.514420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514402 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tls-certs\" (UniqueName: \"kubernetes.io/secret/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tls-certs\") pod \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " Nov 02 08:45:02.514455 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514447 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-dshm\") pod \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " Nov 02 08:45:02.514492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514455 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-home" pod "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" (UID: "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.514524 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514490 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tls-certs\") pod \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " Nov 02 08:45:02.514558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514529 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-dshm\") pod \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " Nov 02 08:45:02.514603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514588 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-home\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-home\") pod \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " Nov 02 08:45:02.514647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514634 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0176aba-5186-4993-84f9-7e5c8b79d1a9-kube-api-access-fmv64\" (UniqueName: \"kubernetes.io/projected/a0176aba-5186-4993-84f9-7e5c8b79d1a9-kube-api-access-fmv64\") pod \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " Nov 02 08:45:02.514708 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514693 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tmp-dir\") pod \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\" (UID: \"a0176aba-5186-4993-84f9-7e5c8b79d1a9\") " Nov 02 08:45:02.514796 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514772 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tmp-dir\") pod \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " Nov 02 08:45:02.514927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.514819 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-model-cache\") pod \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\" (UID: \"af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2\") " Nov 02 08:45:02.515408 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.515280 6160 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-home\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.515512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.515426 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-model-cache" pod "a0176aba-5186-4993-84f9-7e5c8b79d1a9" (UID: "a0176aba-5186-4993-84f9-7e5c8b79d1a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.515512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.515493 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-model-cache" pod "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" (UID: "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.515814 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.515700 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tmp-dir" pod "a0176aba-5186-4993-84f9-7e5c8b79d1a9" (UID: "a0176aba-5186-4993-84f9-7e5c8b79d1a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.515997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.515905 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tmp-dir" pod "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" (UID: "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.515997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.515948 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-home" pod "a0176aba-5186-4993-84f9-7e5c8b79d1a9" (UID: "a0176aba-5186-4993-84f9-7e5c8b79d1a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.517461 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.517427 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-kube-api-access-c8xdl" pod "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" (UID: "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2"). InnerVolumeSpecName "kube-api-access-c8xdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:02.517931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.517905 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-dshm" pod "a0176aba-5186-4993-84f9-7e5c8b79d1a9" (UID: "a0176aba-5186-4993-84f9-7e5c8b79d1a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.518794 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.518752 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tls-certs" pod "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" (UID: "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:02.519083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.519050 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0176aba-5186-4993-84f9-7e5c8b79d1a9-kube-api-access-fmv64" pod "a0176aba-5186-4993-84f9-7e5c8b79d1a9" (UID: "a0176aba-5186-4993-84f9-7e5c8b79d1a9"). InnerVolumeSpecName "kube-api-access-fmv64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:02.519744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.519716 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tls-certs" pod "a0176aba-5186-4993-84f9-7e5c8b79d1a9" (UID: "a0176aba-5186-4993-84f9-7e5c8b79d1a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:02.607240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.607135 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-dshm" pod "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" (UID: "af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:45:02.616267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616238 6160 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-model-cache\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616265 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8xdl\" (UniqueName: \"kubernetes.io/projected/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-kube-api-access-c8xdl\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616278 6160 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tls-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616291 6160 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-dshm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616305 6160 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tls-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616318 6160 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-dshm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616332 6160 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-home\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616341 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmv64\" (UniqueName: \"kubernetes.io/projected/a0176aba-5186-4993-84f9-7e5c8b79d1a9-kube-api-access-fmv64\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616349 6160 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0176aba-5186-4993-84f9-7e5c8b79d1a9-tmp-dir\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616358 6160 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-tmp-dir\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.616473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.616365 6160 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2-model-cache\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:02.885129 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.885046 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt"] Nov 02 08:45:02.887671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.887649 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-2-w2-10d2fb-kserve-5d444b9c75-dfdkt"] Nov 02 08:45:02.987374 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.987337 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8"] Nov 02 08:45:02.991036 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:02.991008 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8"] Nov 02 08:45:03.296114 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:03.296056 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-10d2fb/distinct-w2-10d2fb-kserve-7b9659f7bd-z2mb8" podUID="af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 08:45:03.481355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:03.481317 6160 generic.go:356] "Generic (PLEG): container finished" podID="68726ce2-0477-4746-a217-cc5bb9548622" containerID="5212087243ac3668a4d29c1ffed0ca8507748826160b207d79ef6bc51382ec67" exitCode=60 Nov 02 08:45:03.481873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:03.481407 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" event={"ID":"68726ce2-0477-4746-a217-cc5bb9548622","Type":"ContainerDied","Data":"5212087243ac3668a4d29c1ffed0ca8507748826160b207d79ef6bc51382ec67"} Nov 02 08:45:03.481873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:03.481491 6160 scope.go:122] "RemoveContainer" containerID="9895f82448d30deac2c42f8625d46348d6088aa4ae117edf2dc989cd29c1683d" Nov 02 08:45:03.481873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:03.481810 6160 scope.go:122] "RemoveContainer" containerID="5212087243ac3668a4d29c1ffed0ca8507748826160b207d79ef6bc51382ec67" Nov 02 08:45:03.482136 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:03.482112 6160 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7_odh-ai-gateway-infra(68726ce2-0477-4746-a217-cc5bb9548622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" podUID="68726ce2-0477-4746-a217-cc5bb9548622" Nov 02 08:45:03.585330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:03.585249 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0176aba-5186-4993-84f9-7e5c8b79d1a9" path="/var/lib/kubelet/pods/a0176aba-5186-4993-84f9-7e5c8b79d1a9/volumes" Nov 02 08:45:03.585743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:03.585728 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" path="/var/lib/kubelet/pods/af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2/volumes" Nov 02 08:45:03.632086 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:03.632043 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" podUID="202b4ba4-e975-4a6d-addc-37a40849ef7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:45:03.632273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:03.632058 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" podUID="0434f554-d0b4-4dfb-9d87-efa7039b344c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:45:04.491065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:04.491021 6160 scope.go:122] "RemoveContainer" containerID="5212087243ac3668a4d29c1ffed0ca8507748826160b207d79ef6bc51382ec67" Nov 02 08:45:04.491508 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:04.491310 6160 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7_odh-ai-gateway-infra(68726ce2-0477-4746-a217-cc5bb9548622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" podUID="68726ce2-0477-4746-a217-cc5bb9548622" Nov 02 08:45:04.538494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:04.538464 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm"] Nov 02 08:45:04.649318 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:04.649285 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4"] Nov 02 08:45:04.822828 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:04.822739 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:05.048263 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:05.048204 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:05.234026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:05.233992 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:06.478257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:06.478221 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:09.029379 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.029337 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:09.538953 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.538902 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" podUID="eb26d13b-6288-4a0a-bd62-9d67e929a452" containerName="payload-processing" containerID="cri-o://54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c" gracePeriod=25 Nov 02 08:45:09.649850 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.649802 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" podUID="4faaadd6-625c-4ec0-a989-aed0a8a00ad6" containerName="payload-pre-processing" containerID="cri-o://8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c" gracePeriod=25 Nov 02 08:45:09.808517 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.808456 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:45:09.887306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.887263 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb26d13b-6288-4a0a-bd62-9d67e929a452-kube-api-access-6sdxk\" (UniqueName: \"kubernetes.io/projected/eb26d13b-6288-4a0a-bd62-9d67e929a452-kube-api-access-6sdxk\") pod \"eb26d13b-6288-4a0a-bd62-9d67e929a452\" (UID: \"eb26d13b-6288-4a0a-bd62-9d67e929a452\") " Nov 02 08:45:09.887499 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.887372 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb26d13b-6288-4a0a-bd62-9d67e929a452-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb26d13b-6288-4a0a-bd62-9d67e929a452-plugins-config-volume\") pod \"eb26d13b-6288-4a0a-bd62-9d67e929a452\" (UID: \"eb26d13b-6288-4a0a-bd62-9d67e929a452\") " Nov 02 08:45:09.887747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.887725 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb26d13b-6288-4a0a-bd62-9d67e929a452-plugins-config-volume" pod "eb26d13b-6288-4a0a-bd62-9d67e929a452" (UID: "eb26d13b-6288-4a0a-bd62-9d67e929a452"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:09.889608 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.889584 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb26d13b-6288-4a0a-bd62-9d67e929a452-kube-api-access-6sdxk" pod "eb26d13b-6288-4a0a-bd62-9d67e929a452" (UID: "eb26d13b-6288-4a0a-bd62-9d67e929a452"). InnerVolumeSpecName "kube-api-access-6sdxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:09.906233 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.906212 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:45:09.988130 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.988101 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-kube-api-access-7qw68\" (UniqueName: \"kubernetes.io/projected/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-kube-api-access-7qw68\") pod \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\" (UID: \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\") " Nov 02 08:45:09.988300 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.988260 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-plugins-config-volume\") pod \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\" (UID: \"4faaadd6-625c-4ec0-a989-aed0a8a00ad6\") " Nov 02 08:45:09.988542 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.988495 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdxk\" (UniqueName: \"kubernetes.io/projected/eb26d13b-6288-4a0a-bd62-9d67e929a452-kube-api-access-6sdxk\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:09.988542 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.988509 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb26d13b-6288-4a0a-bd62-9d67e929a452-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:09.988703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.988604 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-plugins-config-volume" pod "4faaadd6-625c-4ec0-a989-aed0a8a00ad6" (UID: "4faaadd6-625c-4ec0-a989-aed0a8a00ad6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:09.990342 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:09.990321 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-kube-api-access-7qw68" pod "4faaadd6-625c-4ec0-a989-aed0a8a00ad6" (UID: "4faaadd6-625c-4ec0-a989-aed0a8a00ad6"). InnerVolumeSpecName "kube-api-access-7qw68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:10.089437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.089352 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qw68\" (UniqueName: \"kubernetes.io/projected/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-kube-api-access-7qw68\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:10.089437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.089382 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faaadd6-625c-4ec0-a989-aed0a8a00ad6-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:10.521541 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:10.521503 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:45:10.529234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.529199 6160 generic.go:356] "Generic (PLEG): container finished" podID="4faaadd6-625c-4ec0-a989-aed0a8a00ad6" containerID="8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c" exitCode=0 Nov 02 08:45:10.529414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.529252 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" Nov 02 08:45:10.529414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.529291 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" event={"ID":"4faaadd6-625c-4ec0-a989-aed0a8a00ad6","Type":"ContainerDied","Data":"8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c"} Nov 02 08:45:10.529414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.529347 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4" event={"ID":"4faaadd6-625c-4ec0-a989-aed0a8a00ad6","Type":"ContainerDied","Data":"ef039993bc386f41845fe93d3f11522e1e025b0b09ec2bee344f4926c7a88f60"} Nov 02 08:45:10.529414 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.529375 6160 scope.go:122] "RemoveContainer" containerID="8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c" Nov 02 08:45:10.530711 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.530692 6160 generic.go:356] "Generic (PLEG): container finished" podID="eb26d13b-6288-4a0a-bd62-9d67e929a452" containerID="54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c" exitCode=0 Nov 02 08:45:10.530829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.530721 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" event={"ID":"eb26d13b-6288-4a0a-bd62-9d67e929a452","Type":"ContainerDied","Data":"54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c"} Nov 02 08:45:10.530829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.530742 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" Nov 02 08:45:10.530829 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.530753 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm" event={"ID":"eb26d13b-6288-4a0a-bd62-9d67e929a452","Type":"ContainerDied","Data":"ef7bc768162562cc96898f948dd02d7c010404cd56158bd7a82ca973dbd8c555"} Nov 02 08:45:10.541091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.540951 6160 scope.go:122] "RemoveContainer" containerID="8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c" Nov 02 08:45:10.541290 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:10.541273 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c\": container with ID starting with 8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c not found: ID does not exist" containerID="8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c" Nov 02 08:45:10.541333 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.541299 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c"} err="failed to get container status \"8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c\": rpc error: code = NotFound desc = could not find container \"8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c\": container with ID starting with 8c8677d9118248f2af42bd764a634ba7f45de276f34a5a6df3d43857076c1e4c not found: ID does not exist" Nov 02 08:45:10.541333 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.541315 6160 scope.go:122] "RemoveContainer" containerID="54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c" Nov 02 08:45:10.562493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.562473 6160 scope.go:122] "RemoveContainer" containerID="54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c" Nov 02 08:45:10.562816 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:10.562796 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c\": container with ID starting with 54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c not found: ID does not exist" containerID="54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c" Nov 02 08:45:10.562869 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.562826 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c"} err="failed to get container status \"54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c\": rpc error: code = NotFound desc = could not find container \"54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c\": container with ID starting with 54bc716a993776fe2e28e278b81c87a2e3853132f783cbb931a793dc3d89533c not found: ID does not exist" Nov 02 08:45:10.685148 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.685115 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm"] Nov 02 08:45:10.732052 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.732001 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-b7bf7884c-qt6qm"] Nov 02 08:45:10.734924 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.734896 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv"] Nov 02 08:45:10.735196 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.735128 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" podUID="f4b219eb-aa01-480c-b92b-8ce43fff3718" containerName="maas-api" containerID="cri-o://0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335" gracePeriod=30 Nov 02 08:45:10.792719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.792629 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4"] Nov 02 08:45:10.796676 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.796648 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-65669556d7-7l7v4"] Nov 02 08:45:10.820851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.820811 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:10.992085 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:10.992061 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:45:11.100489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.100354 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls\") pod \"f4b219eb-aa01-480c-b92b-8ce43fff3718\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " Nov 02 08:45:11.100489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.100460 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-metrics-certs\") pod \"f4b219eb-aa01-480c-b92b-8ce43fff3718\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " Nov 02 08:45:11.101270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.100497 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b219eb-aa01-480c-b92b-8ce43fff3718-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b219eb-aa01-480c-b92b-8ce43fff3718-openshift-service-ca\") pod \"f4b219eb-aa01-480c-b92b-8ce43fff3718\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " Nov 02 08:45:11.101270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.100534 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b219eb-aa01-480c-b92b-8ce43fff3718-kube-api-access-bmxwb\" (UniqueName: \"kubernetes.io/projected/f4b219eb-aa01-480c-b92b-8ce43fff3718-kube-api-access-bmxwb\") pod \"f4b219eb-aa01-480c-b92b-8ce43fff3718\" (UID: \"f4b219eb-aa01-480c-b92b-8ce43fff3718\") " Nov 02 08:45:11.101270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.100920 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b219eb-aa01-480c-b92b-8ce43fff3718-openshift-service-ca" pod "f4b219eb-aa01-480c-b92b-8ce43fff3718" (UID: "f4b219eb-aa01-480c-b92b-8ce43fff3718"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:11.103428 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.103386 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b219eb-aa01-480c-b92b-8ce43fff3718-kube-api-access-bmxwb" pod "f4b219eb-aa01-480c-b92b-8ce43fff3718" (UID: "f4b219eb-aa01-480c-b92b-8ce43fff3718"). InnerVolumeSpecName "kube-api-access-bmxwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:11.103552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.103446 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-metrics-certs" pod "f4b219eb-aa01-480c-b92b-8ce43fff3718" (UID: "f4b219eb-aa01-480c-b92b-8ce43fff3718"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:11.104855 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.104827 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls" pod "f4b219eb-aa01-480c-b92b-8ce43fff3718" (UID: "f4b219eb-aa01-480c-b92b-8ce43fff3718"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:11.212902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.212741 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:11.212902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.212810 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b219eb-aa01-480c-b92b-8ce43fff3718-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:11.212902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.212831 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b219eb-aa01-480c-b92b-8ce43fff3718-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:11.212902 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.212849 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmxwb\" (UniqueName: \"kubernetes.io/projected/f4b219eb-aa01-480c-b92b-8ce43fff3718-kube-api-access-bmxwb\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:11.539635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.539601 6160 generic.go:356] "Generic (PLEG): container finished" podID="f4b219eb-aa01-480c-b92b-8ce43fff3718" containerID="0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335" exitCode=0 Nov 02 08:45:11.539827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.539660 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" Nov 02 08:45:11.539827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.539685 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" event={"ID":"f4b219eb-aa01-480c-b92b-8ce43fff3718","Type":"ContainerDied","Data":"0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335"} Nov 02 08:45:11.539827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.539760 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv" event={"ID":"f4b219eb-aa01-480c-b92b-8ce43fff3718","Type":"ContainerDied","Data":"cd2482b614431a2eed7a4922d50062d896098b2744f1879abc9808385ab7b844"} Nov 02 08:45:11.539827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.539787 6160 scope.go:122] "RemoveContainer" containerID="0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335" Nov 02 08:45:11.551658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.551489 6160 scope.go:122] "RemoveContainer" containerID="0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335" Nov 02 08:45:11.551824 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:11.551804 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335\": container with ID starting with 0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335 not found: ID does not exist" containerID="0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335" Nov 02 08:45:11.551868 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.551836 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335"} err="failed to get container status \"0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335\": rpc error: code = NotFound desc = could not find container \"0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335\": container with ID starting with 0aebb23afe659885b546974806cd4b052ef4c7e80b12eb73fbcd431e5a689335 not found: ID does not exist" Nov 02 08:45:11.583841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.583810 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4faaadd6-625c-4ec0-a989-aed0a8a00ad6" path="/var/lib/kubelet/pods/4faaadd6-625c-4ec0-a989-aed0a8a00ad6/volumes" Nov 02 08:45:11.584225 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.584212 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb26d13b-6288-4a0a-bd62-9d67e929a452" path="/var/lib/kubelet/pods/eb26d13b-6288-4a0a-bd62-9d67e929a452/volumes" Nov 02 08:45:11.693949 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.693918 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv"] Nov 02 08:45:11.700145 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:11.700114 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e6ca80-5c4d468754-fqtkv"] Nov 02 08:45:12.413572 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:12.413532 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:12.614857 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:12.614817 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:12.766431 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:12.766343 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:13.367919 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:13.367885 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:13.586504 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:13.586470 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b219eb-aa01-480c-b92b-8ce43fff3718" path="/var/lib/kubelet/pods/f4b219eb-aa01-480c-b92b-8ce43fff3718/volumes" Nov 02 08:45:13.632786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:13.632709 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" podUID="0434f554-d0b4-4dfb-9d87-efa7039b344c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:45:13.632786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:13.632745 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" podUID="202b4ba4-e975-4a6d-addc-37a40849ef7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:45:16.328679 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:16.328638 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:17.577495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:17.577460 6160 scope.go:122] "RemoveContainer" containerID="5212087243ac3668a4d29c1ffed0ca8507748826160b207d79ef6bc51382ec67" Nov 02 08:45:18.052139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:18.052102 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:18.593398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:18.593362 6160 generic.go:356] "Generic (PLEG): container finished" podID="68726ce2-0477-4746-a217-cc5bb9548622" containerID="9d6aebf4e09c7d7967d7d1cfd51b7e0cb0cf0c2f0f4eafd7dc074b2df9185646" exitCode=60 Nov 02 08:45:18.593849 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:18.593440 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" event={"ID":"68726ce2-0477-4746-a217-cc5bb9548622","Type":"ContainerDied","Data":"9d6aebf4e09c7d7967d7d1cfd51b7e0cb0cf0c2f0f4eafd7dc074b2df9185646"} Nov 02 08:45:18.593849 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:18.593500 6160 scope.go:122] "RemoveContainer" containerID="5212087243ac3668a4d29c1ffed0ca8507748826160b207d79ef6bc51382ec67" Nov 02 08:45:18.593849 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:18.593844 6160 scope.go:122] "RemoveContainer" containerID="9d6aebf4e09c7d7967d7d1cfd51b7e0cb0cf0c2f0f4eafd7dc074b2df9185646" Nov 02 08:45:18.594166 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:18.594144 6160 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7_odh-ai-gateway-infra(68726ce2-0477-4746-a217-cc5bb9548622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" podUID="68726ce2-0477-4746-a217-cc5bb9548622" Nov 02 08:45:19.449215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.449156 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:45:19.503518 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.503490 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:45:19.518078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.518052 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0434f554-d0b4-4dfb-9d87-efa7039b344c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0434f554-d0b4-4dfb-9d87-efa7039b344c-plugins-config-volume\") pod \"0434f554-d0b4-4dfb-9d87-efa7039b344c\" (UID: \"0434f554-d0b4-4dfb-9d87-efa7039b344c\") " Nov 02 08:45:19.518240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.518161 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0434f554-d0b4-4dfb-9d87-efa7039b344c-kube-api-access-k5ftv\" (UniqueName: \"kubernetes.io/projected/0434f554-d0b4-4dfb-9d87-efa7039b344c-kube-api-access-k5ftv\") pod \"0434f554-d0b4-4dfb-9d87-efa7039b344c\" (UID: \"0434f554-d0b4-4dfb-9d87-efa7039b344c\") " Nov 02 08:45:19.518427 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.518403 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0434f554-d0b4-4dfb-9d87-efa7039b344c-plugins-config-volume" pod "0434f554-d0b4-4dfb-9d87-efa7039b344c" (UID: "0434f554-d0b4-4dfb-9d87-efa7039b344c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:19.520454 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.520435 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0434f554-d0b4-4dfb-9d87-efa7039b344c-kube-api-access-k5ftv" pod "0434f554-d0b4-4dfb-9d87-efa7039b344c" (UID: "0434f554-d0b4-4dfb-9d87-efa7039b344c"). InnerVolumeSpecName "kube-api-access-k5ftv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:19.602127 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.602091 6160 generic.go:356] "Generic (PLEG): container finished" podID="0434f554-d0b4-4dfb-9d87-efa7039b344c" containerID="21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344" exitCode=0 Nov 02 08:45:19.602533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.602130 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" Nov 02 08:45:19.602533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.602202 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" event={"ID":"0434f554-d0b4-4dfb-9d87-efa7039b344c","Type":"ContainerDied","Data":"21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344"} Nov 02 08:45:19.602533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.602258 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8" event={"ID":"0434f554-d0b4-4dfb-9d87-efa7039b344c","Type":"ContainerDied","Data":"10fdf757277b0668798c76caf7cb836897107044c631d1564dbe18732e4b6b5d"} Nov 02 08:45:19.602533 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.602286 6160 scope.go:122] "RemoveContainer" containerID="21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344" Nov 02 08:45:19.605655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.605633 6160 generic.go:356] "Generic (PLEG): container finished" podID="202b4ba4-e975-4a6d-addc-37a40849ef7f" containerID="6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571" exitCode=0 Nov 02 08:45:19.605779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.605678 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" Nov 02 08:45:19.605779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.605694 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" event={"ID":"202b4ba4-e975-4a6d-addc-37a40849ef7f","Type":"ContainerDied","Data":"6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571"} Nov 02 08:45:19.605779 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.605745 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx" event={"ID":"202b4ba4-e975-4a6d-addc-37a40849ef7f","Type":"ContainerDied","Data":"0d46cc48654b1d1afc9ac7ec284dd52cc7a3fd6d4c1597e42078a03653fa62e0"} Nov 02 08:45:19.612493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.612473 6160 scope.go:122] "RemoveContainer" containerID="21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344" Nov 02 08:45:19.612788 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:19.612769 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344\": container with ID starting with 21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344 not found: ID does not exist" containerID="21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344" Nov 02 08:45:19.612863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.612801 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344"} err="failed to get container status \"21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344\": rpc error: code = NotFound desc = could not find container \"21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344\": container with ID starting with 21e1f8e6f6f27d308e62fd7f72a1aa004b4bb5083ac7ba8a5eba05ee06844344 not found: ID does not exist" Nov 02 08:45:19.612863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.612823 6160 scope.go:122] "RemoveContainer" containerID="6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571" Nov 02 08:45:19.619119 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.619099 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202b4ba4-e975-4a6d-addc-37a40849ef7f-kube-api-access-rzfv9\" (UniqueName: \"kubernetes.io/projected/202b4ba4-e975-4a6d-addc-37a40849ef7f-kube-api-access-rzfv9\") pod \"202b4ba4-e975-4a6d-addc-37a40849ef7f\" (UID: \"202b4ba4-e975-4a6d-addc-37a40849ef7f\") " Nov 02 08:45:19.619226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.619156 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202b4ba4-e975-4a6d-addc-37a40849ef7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b4ba4-e975-4a6d-addc-37a40849ef7f-plugins-config-volume\") pod \"202b4ba4-e975-4a6d-addc-37a40849ef7f\" (UID: \"202b4ba4-e975-4a6d-addc-37a40849ef7f\") " Nov 02 08:45:19.619512 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.619496 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0434f554-d0b4-4dfb-9d87-efa7039b344c-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:19.619588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.619512 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5ftv\" (UniqueName: \"kubernetes.io/projected/0434f554-d0b4-4dfb-9d87-efa7039b344c-kube-api-access-k5ftv\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:19.619647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.619588 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202b4ba4-e975-4a6d-addc-37a40849ef7f-plugins-config-volume" pod "202b4ba4-e975-4a6d-addc-37a40849ef7f" (UID: "202b4ba4-e975-4a6d-addc-37a40849ef7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:19.621272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.621243 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202b4ba4-e975-4a6d-addc-37a40849ef7f-kube-api-access-rzfv9" pod "202b4ba4-e975-4a6d-addc-37a40849ef7f" (UID: "202b4ba4-e975-4a6d-addc-37a40849ef7f"). InnerVolumeSpecName "kube-api-access-rzfv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:19.621734 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.621708 6160 scope.go:122] "RemoveContainer" containerID="6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571" Nov 02 08:45:19.622005 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:19.621985 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571\": container with ID starting with 6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571 not found: ID does not exist" containerID="6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571" Nov 02 08:45:19.622072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.622015 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571"} err="failed to get container status \"6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571\": rpc error: code = NotFound desc = could not find container \"6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571\": container with ID starting with 6e8391fdc9a8610a3b5b8a0032fd58444bcf5388b805832212f52580213e7571 not found: ID does not exist" Nov 02 08:45:19.711265 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.711190 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7"] Nov 02 08:45:19.720775 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.720746 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzfv9\" (UniqueName: \"kubernetes.io/projected/202b4ba4-e975-4a6d-addc-37a40849ef7f-kube-api-access-rzfv9\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:19.720775 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.720774 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b4ba4-e975-4a6d-addc-37a40849ef7f-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:19.782215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.780331 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8"] Nov 02 08:45:19.782215 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.781411 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-10d2fb-6bd458c589-cnsb8"] Nov 02 08:45:19.834666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.834642 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:19.897812 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.897782 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:19.923530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.923502 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68726ce2-0477-4746-a217-cc5bb9548622-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68726ce2-0477-4746-a217-cc5bb9548622-maas-api-service-ca\") pod \"68726ce2-0477-4746-a217-cc5bb9548622\" (UID: \"68726ce2-0477-4746-a217-cc5bb9548622\") " Nov 02 08:45:19.923717 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.923615 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68726ce2-0477-4746-a217-cc5bb9548622-kube-api-access-bqfrv\" (UniqueName: \"kubernetes.io/projected/68726ce2-0477-4746-a217-cc5bb9548622-kube-api-access-bqfrv\") pod \"68726ce2-0477-4746-a217-cc5bb9548622\" (UID: \"68726ce2-0477-4746-a217-cc5bb9548622\") " Nov 02 08:45:19.923939 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.923910 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68726ce2-0477-4746-a217-cc5bb9548622-maas-api-service-ca" pod "68726ce2-0477-4746-a217-cc5bb9548622" (UID: "68726ce2-0477-4746-a217-cc5bb9548622"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:19.926568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:19.926534 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68726ce2-0477-4746-a217-cc5bb9548622-kube-api-access-bqfrv" pod "68726ce2-0477-4746-a217-cc5bb9548622" (UID: "68726ce2-0477-4746-a217-cc5bb9548622"). InnerVolumeSpecName "kube-api-access-bqfrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:20.025473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.025384 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqfrv\" (UniqueName: \"kubernetes.io/projected/68726ce2-0477-4746-a217-cc5bb9548622-kube-api-access-bqfrv\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:20.025473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.025413 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68726ce2-0477-4746-a217-cc5bb9548622-maas-api-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:20.039013 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.038978 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx"] Nov 02 08:45:20.043227 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.043168 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-10d2fb-94b576b87-rjztx"] Nov 02 08:45:20.070573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.070541 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:20.242977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.242943 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:20.615965 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.615933 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" Nov 02 08:45:20.616481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.615971 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7" event={"ID":"68726ce2-0477-4746-a217-cc5bb9548622","Type":"ContainerDied","Data":"fbccbc43c2910e14af5168dcf1d73d6354ae5ef6e4b4b2ed426997006c7bbce9"} Nov 02 08:45:20.616481 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.616034 6160 scope.go:122] "RemoveContainer" containerID="9d6aebf4e09c7d7967d7d1cfd51b7e0cb0cf0c2f0f4eafd7dc074b2df9185646" Nov 02 08:45:20.767026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.766991 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7"] Nov 02 08:45:20.813073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:20.813038 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2d5a5a-29893485-pdct7"] Nov 02 08:45:21.583655 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:21.583621 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0434f554-d0b4-4dfb-9d87-efa7039b344c" path="/var/lib/kubelet/pods/0434f554-d0b4-4dfb-9d87-efa7039b344c/volumes" Nov 02 08:45:21.584026 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:21.584013 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202b4ba4-e975-4a6d-addc-37a40849ef7f" path="/var/lib/kubelet/pods/202b4ba4-e975-4a6d-addc-37a40849ef7f/volumes" Nov 02 08:45:21.584383 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:21.584370 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68726ce2-0477-4746-a217-cc5bb9548622" path="/var/lib/kubelet/pods/68726ce2-0477-4746-a217-cc5bb9548622/volumes" Nov 02 08:45:23.942545 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:23.942508 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:24.105240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:24.105209 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:24.609905 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:24.609858 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:24.768934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:24.768895 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:25.628704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:25.628665 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:29.692622 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:29.692590 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:30.362801 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:30.362765 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:30.520227 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:30.520146 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:31.849720 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:31.849680 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:32.057911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:32.057879 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:33.265477 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:33.265440 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:33.759672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:33.759633 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:34.843693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.843658 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7"] Nov 02 08:45:34.844142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844129 6160 cpu_manager.go:414] "containerMap: removing container" podUID="0434f554-d0b4-4dfb-9d87-efa7039b344c" containerName="payload-processing" Nov 02 08:45:34.844214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844143 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0434f554-d0b4-4dfb-9d87-efa7039b344c" containerName="payload-processing" Nov 02 08:45:34.844214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844153 6160 cpu_manager.go:414] "containerMap: removing container" podUID="68726ce2-0477-4746-a217-cc5bb9548622" containerName="cleanup" Nov 02 08:45:34.844214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844158 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68726ce2-0477-4746-a217-cc5bb9548622" containerName="cleanup" Nov 02 08:45:34.844214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844166 6160 cpu_manager.go:414] "containerMap: removing container" podUID="68726ce2-0477-4746-a217-cc5bb9548622" containerName="cleanup" Nov 02 08:45:34.844214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844184 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68726ce2-0477-4746-a217-cc5bb9548622" containerName="cleanup" Nov 02 08:45:34.844214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844194 6160 cpu_manager.go:414] "containerMap: removing container" podUID="68726ce2-0477-4746-a217-cc5bb9548622" containerName="cleanup" Nov 02 08:45:34.844214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844199 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68726ce2-0477-4746-a217-cc5bb9548622" containerName="cleanup" Nov 02 08:45:34.844214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844207 6160 cpu_manager.go:414] "containerMap: removing container" podUID="4faaadd6-625c-4ec0-a989-aed0a8a00ad6" containerName="payload-pre-processing" Nov 02 08:45:34.844214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844212 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4faaadd6-625c-4ec0-a989-aed0a8a00ad6" containerName="payload-pre-processing" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844231 6160 cpu_manager.go:414] "containerMap: removing container" podUID="202b4ba4-e975-4a6d-addc-37a40849ef7f" containerName="payload-pre-processing" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844237 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202b4ba4-e975-4a6d-addc-37a40849ef7f" containerName="payload-pre-processing" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844250 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a0176aba-5186-4993-84f9-7e5c8b79d1a9" containerName="main" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844255 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0176aba-5186-4993-84f9-7e5c8b79d1a9" containerName="main" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844262 6160 cpu_manager.go:414] "containerMap: removing container" podUID="f4b219eb-aa01-480c-b92b-8ce43fff3718" containerName="maas-api" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844266 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b219eb-aa01-480c-b92b-8ce43fff3718" containerName="maas-api" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844276 6160 cpu_manager.go:414] "containerMap: removing container" podUID="af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" containerName="main" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844281 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" containerName="main" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844289 6160 cpu_manager.go:414] "containerMap: removing container" podUID="eb26d13b-6288-4a0a-bd62-9d67e929a452" containerName="payload-processing" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844294 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb26d13b-6288-4a0a-bd62-9d67e929a452" containerName="payload-processing" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844382 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="0434f554-d0b4-4dfb-9d87-efa7039b344c" containerName="payload-processing" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844394 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="68726ce2-0477-4746-a217-cc5bb9548622" containerName="cleanup" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844404 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="68726ce2-0477-4746-a217-cc5bb9548622" containerName="cleanup" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844414 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="68726ce2-0477-4746-a217-cc5bb9548622" containerName="cleanup" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844422 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="4faaadd6-625c-4ec0-a989-aed0a8a00ad6" containerName="payload-pre-processing" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844432 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="202b4ba4-e975-4a6d-addc-37a40849ef7f" containerName="payload-pre-processing" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844440 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0176aba-5186-4993-84f9-7e5c8b79d1a9" containerName="main" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844446 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b219eb-aa01-480c-b92b-8ce43fff3718" containerName="maas-api" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844451 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4a79e8-f9f5-4688-a58e-5c02fc2c0ca2" containerName="main" Nov 02 08:45:34.844505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.844457 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb26d13b-6288-4a0a-bd62-9d67e929a452" containerName="payload-processing" Nov 02 08:45:34.847672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.847655 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:34.874267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.874240 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83079646-b7f8-4ccc-bac2-3e1b26fa630c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7\" (UID: \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:34.874415 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.874374 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lck6z\" (UniqueName: \"kubernetes.io/projected/83079646-b7f8-4ccc-bac2-3e1b26fa630c-kube-api-access-lck6z\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7\" (UID: \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:34.942330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.942294 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d91f6f-dockercfg-nj5st\"" Nov 02 08:45:34.942500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.942364 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d91f6f\"" Nov 02 08:45:34.944923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.944904 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7"] Nov 02 08:45:34.975433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.975399 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83079646-b7f8-4ccc-bac2-3e1b26fa630c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7\" (UID: \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:34.975600 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.975480 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lck6z\" (UniqueName: \"kubernetes.io/projected/83079646-b7f8-4ccc-bac2-3e1b26fa630c-kube-api-access-lck6z\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7\" (UID: \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:34.976091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:34.976070 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83079646-b7f8-4ccc-bac2-3e1b26fa630c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7\" (UID: \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:35.028831 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.028797 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lck6z\" (UniqueName: \"kubernetes.io/projected/83079646-b7f8-4ccc-bac2-3e1b26fa630c-kube-api-access-lck6z\") pod \"payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7\" (UID: \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:35.161076 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.160991 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:35.527809 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.527779 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7"] Nov 02 08:45:35.529243 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:45:35.529215 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83079646_b7f8_4ccc_bac2_3e1b26fa630c.slice/crio-cd6acf4091c2b612947c0af85c241d636ffc1a2ed66e20e04ce2cff1e62d7f06 WatchSource:0}: Error finding container cd6acf4091c2b612947c0af85c241d636ffc1a2ed66e20e04ce2cff1e62d7f06: Status 404 returned error can't find the container with id cd6acf4091c2b612947c0af85c241d636ffc1a2ed66e20e04ce2cff1e62d7f06 Nov 02 08:45:35.640207 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.640161 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk"] Nov 02 08:45:35.644310 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.644286 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:35.682003 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.681972 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kpkn\" (UniqueName: \"kubernetes.io/projected/c6c0574a-cf84-4f48-9e56-15da5225e58d-kube-api-access-9kpkn\") pod \"payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk\" (UID: \"c6c0574a-cf84-4f48-9e56-15da5225e58d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:35.682198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.682011 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c0574a-cf84-4f48-9e56-15da5225e58d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk\" (UID: \"c6c0574a-cf84-4f48-9e56-15da5225e58d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:35.698467 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.698438 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk"] Nov 02 08:45:35.726456 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.726420 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" event={"ID":"83079646-b7f8-4ccc-bac2-3e1b26fa630c","Type":"ContainerStarted","Data":"a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4"} Nov 02 08:45:35.726640 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.726509 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:35.726640 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.726527 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" event={"ID":"83079646-b7f8-4ccc-bac2-3e1b26fa630c","Type":"ContainerStarted","Data":"cd6acf4091c2b612947c0af85c241d636ffc1a2ed66e20e04ce2cff1e62d7f06"} Nov 02 08:45:35.783108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.783025 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kpkn\" (UniqueName: \"kubernetes.io/projected/c6c0574a-cf84-4f48-9e56-15da5225e58d-kube-api-access-9kpkn\") pod \"payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk\" (UID: \"c6c0574a-cf84-4f48-9e56-15da5225e58d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:35.783108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.783071 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c0574a-cf84-4f48-9e56-15da5225e58d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk\" (UID: \"c6c0574a-cf84-4f48-9e56-15da5225e58d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:35.783850 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.783824 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c0574a-cf84-4f48-9e56-15da5225e58d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk\" (UID: \"c6c0574a-cf84-4f48-9e56-15da5225e58d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:35.837737 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.837702 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kpkn\" (UniqueName: \"kubernetes.io/projected/c6c0574a-cf84-4f48-9e56-15da5225e58d-kube-api-access-9kpkn\") pod \"payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk\" (UID: \"c6c0574a-cf84-4f48-9e56-15da5225e58d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:35.861815 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.861758 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" podStartSLOduration=1.861742937 podStartE2EDuration="1.861742937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:35.859964538 +0000 UTC m=+2334.413468747" watchObservedRunningTime="2026-11-02 08:45:35.861742937 +0000 UTC m=+2334.415247132" Nov 02 08:45:35.957814 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:35.957777 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:36.334007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:36.333974 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk"] Nov 02 08:45:36.334213 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:45:36.334188 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c0574a_cf84_4f48_9e56_15da5225e58d.slice/crio-bb10d8fdbddb761125c8adc67f8dc9dfd86123d1a8491892cdaaf93325109349 WatchSource:0}: Error finding container bb10d8fdbddb761125c8adc67f8dc9dfd86123d1a8491892cdaaf93325109349: Status 404 returned error can't find the container with id bb10d8fdbddb761125c8adc67f8dc9dfd86123d1a8491892cdaaf93325109349 Nov 02 08:45:36.343885 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:36.343843 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:36.736585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:36.736551 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" event={"ID":"c6c0574a-cf84-4f48-9e56-15da5225e58d","Type":"ContainerStarted","Data":"2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da"} Nov 02 08:45:36.736585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:36.736606 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" event={"ID":"c6c0574a-cf84-4f48-9e56-15da5225e58d","Type":"ContainerStarted","Data":"bb10d8fdbddb761125c8adc67f8dc9dfd86123d1a8491892cdaaf93325109349"} Nov 02 08:45:36.736883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:36.736785 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:36.842548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:36.842484 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" podStartSLOduration=1.8424645339999999 podStartE2EDuration="1.842464534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:36.842260656 +0000 UTC m=+2335.395764879" watchObservedRunningTime="2026-11-02 08:45:36.842464534 +0000 UTC m=+2335.395968732" Nov 02 08:45:40.521323 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:40.521288 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:45:41.231189 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:41.231135 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:46.739259 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:46.739229 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:45:46.941872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:46.941839 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc"] Nov 02 08:45:46.942067 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:46.942045 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" podUID="d81bdb8e-3bce-4589-b74e-29be4446dd50" containerName="maas-api" containerID="cri-o://e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8" gracePeriod=30 Nov 02 08:45:47.086775 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.086742 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:47.201186 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.201138 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:45:47.303281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.303250 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81bdb8e-3bce-4589-b74e-29be4446dd50-kube-api-access-g5hln\" (UniqueName: \"kubernetes.io/projected/d81bdb8e-3bce-4589-b74e-29be4446dd50-kube-api-access-g5hln\") pod \"d81bdb8e-3bce-4589-b74e-29be4446dd50\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " Nov 02 08:45:47.303454 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.303335 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls\") pod \"d81bdb8e-3bce-4589-b74e-29be4446dd50\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " Nov 02 08:45:47.303454 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.303390 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-metrics-certs\") pod \"d81bdb8e-3bce-4589-b74e-29be4446dd50\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " Nov 02 08:45:47.303454 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.303447 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81bdb8e-3bce-4589-b74e-29be4446dd50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81bdb8e-3bce-4589-b74e-29be4446dd50-openshift-service-ca\") pod \"d81bdb8e-3bce-4589-b74e-29be4446dd50\" (UID: \"d81bdb8e-3bce-4589-b74e-29be4446dd50\") " Nov 02 08:45:47.303884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.303854 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81bdb8e-3bce-4589-b74e-29be4446dd50-openshift-service-ca" pod "d81bdb8e-3bce-4589-b74e-29be4446dd50" (UID: "d81bdb8e-3bce-4589-b74e-29be4446dd50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:47.305582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.305548 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-metrics-certs" pod "d81bdb8e-3bce-4589-b74e-29be4446dd50" (UID: "d81bdb8e-3bce-4589-b74e-29be4446dd50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:47.305582 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.305567 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81bdb8e-3bce-4589-b74e-29be4446dd50-kube-api-access-g5hln" pod "d81bdb8e-3bce-4589-b74e-29be4446dd50" (UID: "d81bdb8e-3bce-4589-b74e-29be4446dd50"). InnerVolumeSpecName "kube-api-access-g5hln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:47.305729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.305592 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls" pod "d81bdb8e-3bce-4589-b74e-29be4446dd50" (UID: "d81bdb8e-3bce-4589-b74e-29be4446dd50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:47.404963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.404923 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:47.404963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.404957 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81bdb8e-3bce-4589-b74e-29be4446dd50-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:47.404963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.404966 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81bdb8e-3bce-4589-b74e-29be4446dd50-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:47.405267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.404975 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5hln\" (UniqueName: \"kubernetes.io/projected/d81bdb8e-3bce-4589-b74e-29be4446dd50-kube-api-access-g5hln\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:45:47.747142 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.747114 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:45:47.808448 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.808411 6160 generic.go:356] "Generic (PLEG): container finished" podID="d81bdb8e-3bce-4589-b74e-29be4446dd50" containerID="e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8" exitCode=0 Nov 02 08:45:47.808624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.808472 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" Nov 02 08:45:47.808624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.808475 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" event={"ID":"d81bdb8e-3bce-4589-b74e-29be4446dd50","Type":"ContainerDied","Data":"e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8"} Nov 02 08:45:47.808624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.808545 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc" event={"ID":"d81bdb8e-3bce-4589-b74e-29be4446dd50","Type":"ContainerDied","Data":"ee122800fb30dc60d95253288445949eea163b1c1303667139c9925ec8f88df5"} Nov 02 08:45:47.808624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.808578 6160 scope.go:122] "RemoveContainer" containerID="e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8" Nov 02 08:45:47.818584 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.818562 6160 scope.go:122] "RemoveContainer" containerID="e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8" Nov 02 08:45:47.818908 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:45:47.818886 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8\": container with ID starting with e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8 not found: ID does not exist" containerID="e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8" Nov 02 08:45:47.819014 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.818914 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8"} err="failed to get container status \"e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8\": rpc error: code = NotFound desc = could not find container \"e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8\": container with ID starting with e45c8edd8f4395404678e0f00b8e74d0fa5fe5f08a65278049a698cd56c4e7a8 not found: ID does not exist" Nov 02 08:45:47.961395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.961364 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc"] Nov 02 08:45:47.963766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:47.963742 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-10d2fb-b49d8d996-qhwhc"] Nov 02 08:45:48.782938 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:48.782910 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:49.017305 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:49.017267 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:49.188583 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:49.188550 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:49.592601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:49.592523 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81bdb8e-3bce-4589-b74e-29be4446dd50" path="/var/lib/kubelet/pods/d81bdb8e-3bce-4589-b74e-29be4446dd50/volumes" Nov 02 08:45:49.633375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:49.633345 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:55.322300 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:55.322208 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:56.591399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:45:56.591353 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:00.949523 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:00.949486 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:04.979322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:04.979291 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-4h2lv_c7297abc-ee50-4bd9-8065-99c7cd95357f/payload-processing/0.log" Nov 02 08:46:05.228863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:05.228836 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-4h2lv_c7297abc-ee50-4bd9-8065-99c7cd95357f/payload-processing/0.log" Nov 02 08:46:05.468500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:05.468468 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk_c6c0574a-cf84-4f48-9e56-15da5225e58d/payload-processing/0.log" Nov 02 08:46:07.605658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:07.605617 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:09.361234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:09.361197 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:10.520766 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:46:10.520723 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:46:12.746369 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:12.746328 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:18.216304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:18.216271 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr_a9d93ffc-c954-4760-8606-29cf9767c36b/payload-processing/0.log" Nov 02 08:46:18.506256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:18.506147 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-4h2lv_c7297abc-ee50-4bd9-8065-99c7cd95357f/payload-processing/0.log" Nov 02 08:46:18.747647 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:18.747621 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-4h2lv_c7297abc-ee50-4bd9-8065-99c7cd95357f/payload-processing/0.log" Nov 02 08:46:19.152094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:19.152055 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:20.838848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:20.838812 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:20.994458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:20.994422 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:24.589105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:24.589062 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:25.307686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:25.307647 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:25.957692 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:25.957652 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:30.553011 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.552975 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5"] Nov 02 08:46:30.553470 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.553234 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" podUID="9b8d2324-5246-4559-81d8-507723e7ab6b" containerName="maas-api" containerID="cri-o://7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444" gracePeriod=30 Nov 02 08:46:30.680916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.680884 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:30.817034 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.816964 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:46:30.975835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.975786 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8d2324-5246-4559-81d8-507723e7ab6b-kube-api-access-g7nhw\" (UniqueName: \"kubernetes.io/projected/9b8d2324-5246-4559-81d8-507723e7ab6b-kube-api-access-g7nhw\") pod \"9b8d2324-5246-4559-81d8-507723e7ab6b\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " Nov 02 08:46:30.976030 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.975950 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls\") pod \"9b8d2324-5246-4559-81d8-507723e7ab6b\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " Nov 02 08:46:30.976085 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.976037 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8d2324-5246-4559-81d8-507723e7ab6b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2324-5246-4559-81d8-507723e7ab6b-openshift-service-ca\") pod \"9b8d2324-5246-4559-81d8-507723e7ab6b\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " Nov 02 08:46:30.976127 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.976097 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-metrics-certs\") pod \"9b8d2324-5246-4559-81d8-507723e7ab6b\" (UID: \"9b8d2324-5246-4559-81d8-507723e7ab6b\") " Nov 02 08:46:30.976755 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.976720 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8d2324-5246-4559-81d8-507723e7ab6b-openshift-service-ca" pod "9b8d2324-5246-4559-81d8-507723e7ab6b" (UID: "9b8d2324-5246-4559-81d8-507723e7ab6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:46:30.978830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.978805 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls" pod "9b8d2324-5246-4559-81d8-507723e7ab6b" (UID: "9b8d2324-5246-4559-81d8-507723e7ab6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:46:30.979311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.979289 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-metrics-certs" pod "9b8d2324-5246-4559-81d8-507723e7ab6b" (UID: "9b8d2324-5246-4559-81d8-507723e7ab6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:46:30.979424 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:30.979328 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8d2324-5246-4559-81d8-507723e7ab6b-kube-api-access-g7nhw" pod "9b8d2324-5246-4559-81d8-507723e7ab6b" (UID: "9b8d2324-5246-4559-81d8-507723e7ab6b"). InnerVolumeSpecName "kube-api-access-g7nhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:46:31.068552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.068468 6160 generic.go:356] "Generic (PLEG): container finished" podID="9b8d2324-5246-4559-81d8-507723e7ab6b" containerID="7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444" exitCode=0 Nov 02 08:46:31.068552 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.068511 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" Nov 02 08:46:31.068752 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.068554 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" event={"ID":"9b8d2324-5246-4559-81d8-507723e7ab6b","Type":"ContainerDied","Data":"7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444"} Nov 02 08:46:31.068752 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.068604 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5" event={"ID":"9b8d2324-5246-4559-81d8-507723e7ab6b","Type":"ContainerDied","Data":"a03bb644056fc0470bba7c3dd8f94b6270e2a8f0b2fa3464da1dc20e1694a818"} Nov 02 08:46:31.068752 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.068625 6160 scope.go:122] "RemoveContainer" containerID="7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444" Nov 02 08:46:31.077249 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.077225 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:31.077366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.077252 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2324-5246-4559-81d8-507723e7ab6b-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:31.077366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.077266 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b8d2324-5246-4559-81d8-507723e7ab6b-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:31.077366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.077275 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7nhw\" (UniqueName: \"kubernetes.io/projected/9b8d2324-5246-4559-81d8-507723e7ab6b-kube-api-access-g7nhw\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:31.079596 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.079580 6160 scope.go:122] "RemoveContainer" containerID="7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444" Nov 02 08:46:31.079890 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:46:31.079869 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444\": container with ID starting with 7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444 not found: ID does not exist" containerID="7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444" Nov 02 08:46:31.079966 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.079898 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444"} err="failed to get container status \"7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444\": rpc error: code = NotFound desc = could not find container \"7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444\": container with ID starting with 7e1a9c06973e0b6ac370c6187e44d9122c9176d8f6e8612410b8f3c64b71f444 not found: ID does not exist" Nov 02 08:46:31.086761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.086738 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n"] Nov 02 08:46:31.086945 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.086927 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" podUID="b20d3925-caaa-46ae-a508-808eb42aac2e" containerName="payload-pre-processing" containerID="cri-o://c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c" gracePeriod=30 Nov 02 08:46:31.210301 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.210270 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:31.222355 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.222328 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5"] Nov 02 08:46:31.226996 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.226966 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceb40-7dbff657bc-hnth5"] Nov 02 08:46:31.339606 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.339579 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:46:31.427041 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.427007 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:31.482366 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.482331 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20d3925-caaa-46ae-a508-808eb42aac2e-kube-api-access-g9hch\" (UniqueName: \"kubernetes.io/projected/b20d3925-caaa-46ae-a508-808eb42aac2e-kube-api-access-g9hch\") pod \"b20d3925-caaa-46ae-a508-808eb42aac2e\" (UID: \"b20d3925-caaa-46ae-a508-808eb42aac2e\") " Nov 02 08:46:31.482568 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.482475 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20d3925-caaa-46ae-a508-808eb42aac2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b20d3925-caaa-46ae-a508-808eb42aac2e-plugins-config-volume\") pod \"b20d3925-caaa-46ae-a508-808eb42aac2e\" (UID: \"b20d3925-caaa-46ae-a508-808eb42aac2e\") " Nov 02 08:46:31.482867 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.482842 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20d3925-caaa-46ae-a508-808eb42aac2e-plugins-config-volume" pod "b20d3925-caaa-46ae-a508-808eb42aac2e" (UID: "b20d3925-caaa-46ae-a508-808eb42aac2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:46:31.484632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.484611 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20d3925-caaa-46ae-a508-808eb42aac2e-kube-api-access-g9hch" pod "b20d3925-caaa-46ae-a508-808eb42aac2e" (UID: "b20d3925-caaa-46ae-a508-808eb42aac2e"). InnerVolumeSpecName "kube-api-access-g9hch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:46:31.583298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.583225 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b20d3925-caaa-46ae-a508-808eb42aac2e-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:31.583298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.583252 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9hch\" (UniqueName: \"kubernetes.io/projected/b20d3925-caaa-46ae-a508-808eb42aac2e-kube-api-access-g9hch\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:46:31.583866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:31.583845 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8d2324-5246-4559-81d8-507723e7ab6b" path="/var/lib/kubelet/pods/9b8d2324-5246-4559-81d8-507723e7ab6b/volumes" Nov 02 08:46:32.075903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.075868 6160 generic.go:356] "Generic (PLEG): container finished" podID="b20d3925-caaa-46ae-a508-808eb42aac2e" containerID="c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c" exitCode=0 Nov 02 08:46:32.076087 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.075915 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" Nov 02 08:46:32.076087 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.075958 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" event={"ID":"b20d3925-caaa-46ae-a508-808eb42aac2e","Type":"ContainerDied","Data":"c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c"} Nov 02 08:46:32.076087 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.076025 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n" event={"ID":"b20d3925-caaa-46ae-a508-808eb42aac2e","Type":"ContainerDied","Data":"692c3ada4dbffaf13b49555339a038241e8db9b744d9edde7ae71e2903de8965"} Nov 02 08:46:32.076087 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.076054 6160 scope.go:122] "RemoveContainer" containerID="c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c" Nov 02 08:46:32.086272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.086251 6160 scope.go:122] "RemoveContainer" containerID="c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c" Nov 02 08:46:32.086551 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:46:32.086533 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c\": container with ID starting with c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c not found: ID does not exist" containerID="c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c" Nov 02 08:46:32.086592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.086560 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c"} err="failed to get container status \"c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c\": rpc error: code = NotFound desc = could not find container \"c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c\": container with ID starting with c48be15eecda6ea8b427852d14bce0ec492f05afafb9efce81a8738a6125416c not found: ID does not exist" Nov 02 08:46:32.230007 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.229969 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n"] Nov 02 08:46:32.276222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.276165 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceb40-5dd8d5d8d7-4zp6n"] Nov 02 08:46:32.417225 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.417188 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:32.618489 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.618439 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:32.769780 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:32.769691 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:33.267468 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:33.267438 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:33.584282 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:33.584202 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20d3925-caaa-46ae-a508-808eb42aac2e" path="/var/lib/kubelet/pods/b20d3925-caaa-46ae-a508-808eb42aac2e/volumes" Nov 02 08:46:37.834530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:37.834493 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:39.186004 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:39.185957 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:39.396905 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:39.396872 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:39.559963 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:39.559869 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:40.520323 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:46:40.520289 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:46:42.005399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:42.005369 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:46:42.011456 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:42.011434 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:46:42.988621 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:42.988581 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:49.538332 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:49.538287 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:54.445692 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:46:54.445643 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:00.933204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:00.933142 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:05.674957 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:05.674921 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:09.695117 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:09.695082 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq"] Nov 02 08:47:09.695733 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:09.695328 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" podUID="96b483ec-087c-4c10-973c-fea9a66a1073" containerName="maas-api" containerID="cri-o://bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8" gracePeriod=30 Nov 02 08:47:09.735270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:09.735229 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:09.920653 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:09.920613 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:10.156415 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.156328 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr"] Nov 02 08:47:10.156579 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.156558 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" podUID="a9d93ffc-c954-4760-8606-29cf9767c36b" containerName="payload-processing" containerID="cri-o://bbebd5c977381981e40070b08f86c66eb5f20483b2afe28bb6db5bba607273bd" gracePeriod=30 Nov 02 08:47:10.210260 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.210226 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf"] Nov 02 08:47:10.210420 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.210402 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" podUID="5bb4d9c6-5abb-4536-a36e-50390e6dabf0" containerName="payload-pre-processing" containerID="cri-o://fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56" gracePeriod=30 Nov 02 08:47:10.305935 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.305901 6160 generic.go:356] "Generic (PLEG): container finished" podID="a9d93ffc-c954-4760-8606-29cf9767c36b" containerID="bbebd5c977381981e40070b08f86c66eb5f20483b2afe28bb6db5bba607273bd" exitCode=0 Nov 02 08:47:10.306122 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.305980 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" event={"ID":"a9d93ffc-c954-4760-8606-29cf9767c36b","Type":"ContainerDied","Data":"bbebd5c977381981e40070b08f86c66eb5f20483b2afe28bb6db5bba607273bd"} Nov 02 08:47:10.360202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.355194 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:10.484925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.484891 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:47:10.503666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.503636 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d93ffc-c954-4760-8606-29cf9767c36b-kube-api-access-x7fxz\" (UniqueName: \"kubernetes.io/projected/a9d93ffc-c954-4760-8606-29cf9767c36b-kube-api-access-x7fxz\") pod \"a9d93ffc-c954-4760-8606-29cf9767c36b\" (UID: \"a9d93ffc-c954-4760-8606-29cf9767c36b\") " Nov 02 08:47:10.503805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.503699 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d93ffc-c954-4760-8606-29cf9767c36b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d93ffc-c954-4760-8606-29cf9767c36b-plugins-config-volume\") pod \"a9d93ffc-c954-4760-8606-29cf9767c36b\" (UID: \"a9d93ffc-c954-4760-8606-29cf9767c36b\") " Nov 02 08:47:10.504080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.504058 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d93ffc-c954-4760-8606-29cf9767c36b-plugins-config-volume" pod "a9d93ffc-c954-4760-8606-29cf9767c36b" (UID: "a9d93ffc-c954-4760-8606-29cf9767c36b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:47:10.505728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.505706 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d93ffc-c954-4760-8606-29cf9767c36b-kube-api-access-x7fxz" pod "a9d93ffc-c954-4760-8606-29cf9767c36b" (UID: "a9d93ffc-c954-4760-8606-29cf9767c36b"). InnerVolumeSpecName "kube-api-access-x7fxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:47:10.513907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.513881 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:47:10.522982 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:47:10.522954 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:47:10.530163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.530137 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:10.605492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.605464 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-kube-api-access-9zkkp\" (UniqueName: \"kubernetes.io/projected/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-kube-api-access-9zkkp\") pod \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\" (UID: \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\") " Nov 02 08:47:10.605672 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.605573 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-plugins-config-volume\") pod \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\" (UID: \"5bb4d9c6-5abb-4536-a36e-50390e6dabf0\") " Nov 02 08:47:10.606002 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.605979 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7fxz\" (UniqueName: \"kubernetes.io/projected/a9d93ffc-c954-4760-8606-29cf9767c36b-kube-api-access-x7fxz\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:10.606083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.606013 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d93ffc-c954-4760-8606-29cf9767c36b-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:10.606083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.605979 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-plugins-config-volume" pod "5bb4d9c6-5abb-4536-a36e-50390e6dabf0" (UID: "5bb4d9c6-5abb-4536-a36e-50390e6dabf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:47:10.607774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.607747 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-kube-api-access-9zkkp" pod "5bb4d9c6-5abb-4536-a36e-50390e6dabf0" (UID: "5bb4d9c6-5abb-4536-a36e-50390e6dabf0"). InnerVolumeSpecName "kube-api-access-9zkkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:47:10.706729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.706650 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:10.706729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.706682 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zkkp\" (UniqueName: \"kubernetes.io/projected/5bb4d9c6-5abb-4536-a36e-50390e6dabf0-kube-api-access-9zkkp\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:10.902805 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.902765 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b"] Nov 02 08:47:10.903010 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:10.902987 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" podUID="c821cfa2-aa32-4fa8-a9b6-f52884cd658a" containerName="maas-api" containerID="cri-o://01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24" gracePeriod=30 Nov 02 08:47:11.018960 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.018927 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:47:11.035379 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.035334 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:11.111802 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.111763 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls\") pod \"96b483ec-087c-4c10-973c-fea9a66a1073\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " Nov 02 08:47:11.111987 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.111869 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b483ec-087c-4c10-973c-fea9a66a1073-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b483ec-087c-4c10-973c-fea9a66a1073-openshift-service-ca\") pod \"96b483ec-087c-4c10-973c-fea9a66a1073\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " Nov 02 08:47:11.112062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.111987 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-metrics-certs\") pod \"96b483ec-087c-4c10-973c-fea9a66a1073\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " Nov 02 08:47:11.112062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.112024 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b483ec-087c-4c10-973c-fea9a66a1073-kube-api-access-8kfhr\" (UniqueName: \"kubernetes.io/projected/96b483ec-087c-4c10-973c-fea9a66a1073-kube-api-access-8kfhr\") pod \"96b483ec-087c-4c10-973c-fea9a66a1073\" (UID: \"96b483ec-087c-4c10-973c-fea9a66a1073\") " Nov 02 08:47:11.112358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.112322 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b483ec-087c-4c10-973c-fea9a66a1073-openshift-service-ca" pod "96b483ec-087c-4c10-973c-fea9a66a1073" (UID: "96b483ec-087c-4c10-973c-fea9a66a1073"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:47:11.112599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.112575 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b483ec-087c-4c10-973c-fea9a66a1073-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.115413 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.115386 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-metrics-certs" pod "96b483ec-087c-4c10-973c-fea9a66a1073" (UID: "96b483ec-087c-4c10-973c-fea9a66a1073"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:47:11.117131 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.117031 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b483ec-087c-4c10-973c-fea9a66a1073-kube-api-access-8kfhr" pod "96b483ec-087c-4c10-973c-fea9a66a1073" (UID: "96b483ec-087c-4c10-973c-fea9a66a1073"). InnerVolumeSpecName "kube-api-access-8kfhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:47:11.117411 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.117228 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls" pod "96b483ec-087c-4c10-973c-fea9a66a1073" (UID: "96b483ec-087c-4c10-973c-fea9a66a1073"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:47:11.160721 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.160687 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:47:11.213372 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.213342 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-kube-api-access-dnql6\" (UniqueName: \"kubernetes.io/projected/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-kube-api-access-dnql6\") pod \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " Nov 02 08:47:11.213567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.213379 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls\") pod \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " Nov 02 08:47:11.213567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.213447 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-openshift-service-ca\") pod \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " Nov 02 08:47:11.213567 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.213473 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-metrics-certs\") pod \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\" (UID: \"c821cfa2-aa32-4fa8-a9b6-f52884cd658a\") " Nov 02 08:47:11.213753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.213734 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.213753 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.213748 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kfhr\" (UniqueName: \"kubernetes.io/projected/96b483ec-087c-4c10-973c-fea9a66a1073-kube-api-access-8kfhr\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.213850 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.213763 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b483ec-087c-4c10-973c-fea9a66a1073-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.213887 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.213843 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-openshift-service-ca" pod "c821cfa2-aa32-4fa8-a9b6-f52884cd658a" (UID: "c821cfa2-aa32-4fa8-a9b6-f52884cd658a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:47:11.215670 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.215642 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-kube-api-access-dnql6" pod "c821cfa2-aa32-4fa8-a9b6-f52884cd658a" (UID: "c821cfa2-aa32-4fa8-a9b6-f52884cd658a"). InnerVolumeSpecName "kube-api-access-dnql6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:47:11.215798 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.215745 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls" pod "c821cfa2-aa32-4fa8-a9b6-f52884cd658a" (UID: "c821cfa2-aa32-4fa8-a9b6-f52884cd658a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:47:11.215798 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.215777 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-metrics-certs" pod "c821cfa2-aa32-4fa8-a9b6-f52884cd658a" (UID: "c821cfa2-aa32-4fa8-a9b6-f52884cd658a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:47:11.313950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.313847 6160 generic.go:356] "Generic (PLEG): container finished" podID="c821cfa2-aa32-4fa8-a9b6-f52884cd658a" containerID="01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24" exitCode=0 Nov 02 08:47:11.313950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.313899 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" Nov 02 08:47:11.314212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.313966 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" event={"ID":"c821cfa2-aa32-4fa8-a9b6-f52884cd658a","Type":"ContainerDied","Data":"01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24"} Nov 02 08:47:11.314212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.314016 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b" event={"ID":"c821cfa2-aa32-4fa8-a9b6-f52884cd658a","Type":"ContainerDied","Data":"f8d1630c46cf0ff68f4ed581d127cd6550df7021d7675747fb3a9164d7faee57"} Nov 02 08:47:11.314212 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.314047 6160 scope.go:122] "RemoveContainer" containerID="01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24" Nov 02 08:47:11.314530 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.314503 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnql6\" (UniqueName: \"kubernetes.io/projected/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-kube-api-access-dnql6\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.314604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.314539 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.314604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.314556 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.314604 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.314573 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c821cfa2-aa32-4fa8-a9b6-f52884cd658a-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.322649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.322593 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" event={"ID":"a9d93ffc-c954-4760-8606-29cf9767c36b","Type":"ContainerDied","Data":"f0e7539f4fb79e01ccaacb81ebef6742ea6ee0fa1a04e0f34604f66e393d5cfe"} Nov 02 08:47:11.322878 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.322855 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr" Nov 02 08:47:11.328933 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.328896 6160 generic.go:356] "Generic (PLEG): container finished" podID="96b483ec-087c-4c10-973c-fea9a66a1073" containerID="bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8" exitCode=0 Nov 02 08:47:11.329088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.329073 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" Nov 02 08:47:11.329370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.329283 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" event={"ID":"96b483ec-087c-4c10-973c-fea9a66a1073","Type":"ContainerDied","Data":"bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8"} Nov 02 08:47:11.329370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.329339 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq" event={"ID":"96b483ec-087c-4c10-973c-fea9a66a1073","Type":"ContainerDied","Data":"eaba2ad8bf6af85488df31a418c48ae1083f77ea706c5c3cfed7939919e1c831"} Nov 02 08:47:11.331658 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.331632 6160 generic.go:356] "Generic (PLEG): container finished" podID="5bb4d9c6-5abb-4536-a36e-50390e6dabf0" containerID="fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56" exitCode=0 Nov 02 08:47:11.331784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.331692 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" event={"ID":"5bb4d9c6-5abb-4536-a36e-50390e6dabf0","Type":"ContainerDied","Data":"fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56"} Nov 02 08:47:11.331784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.331735 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" event={"ID":"5bb4d9c6-5abb-4536-a36e-50390e6dabf0","Type":"ContainerDied","Data":"d6e85893c7964e08b188e9d3dee068fafec0b06db4d80a1f3bc17d236539d75a"} Nov 02 08:47:11.331911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.331816 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf" Nov 02 08:47:11.332514 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.332471 6160 scope.go:122] "RemoveContainer" containerID="01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24" Nov 02 08:47:11.332848 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:47:11.332778 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24\": container with ID starting with 01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24 not found: ID does not exist" containerID="01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24" Nov 02 08:47:11.332848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.332818 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24"} err="failed to get container status \"01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24\": rpc error: code = NotFound desc = could not find container \"01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24\": container with ID starting with 01a0e4b578080850fffedc1144aebd8720375beefa76f131c9cc339d2654df24 not found: ID does not exist" Nov 02 08:47:11.332848 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.332844 6160 scope.go:122] "RemoveContainer" containerID="bbebd5c977381981e40070b08f86c66eb5f20483b2afe28bb6db5bba607273bd" Nov 02 08:47:11.347411 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.347384 6160 scope.go:122] "RemoveContainer" containerID="bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8" Nov 02 08:47:11.357137 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.357112 6160 scope.go:122] "RemoveContainer" containerID="bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8" Nov 02 08:47:11.357790 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:47:11.357508 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8\": container with ID starting with bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8 not found: ID does not exist" containerID="bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8" Nov 02 08:47:11.357790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.357540 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8"} err="failed to get container status \"bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8\": rpc error: code = NotFound desc = could not find container \"bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8\": container with ID starting with bbfe3f91a459d8695619f68b5123c9bd7a5491e7bfac148219ae24d0e02801c8 not found: ID does not exist" Nov 02 08:47:11.357790 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.357567 6160 scope.go:122] "RemoveContainer" containerID="fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56" Nov 02 08:47:11.360398 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.360375 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk"] Nov 02 08:47:11.360599 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.360579 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" podUID="c6c0574a-cf84-4f48-9e56-15da5225e58d" containerName="payload-processing" containerID="cri-o://2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da" gracePeriod=30 Nov 02 08:47:11.370417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.370390 6160 scope.go:122] "RemoveContainer" containerID="fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56" Nov 02 08:47:11.370965 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:47:11.370927 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56\": container with ID starting with fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56 not found: ID does not exist" containerID="fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56" Nov 02 08:47:11.371057 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.370981 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56"} err="failed to get container status \"fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56\": rpc error: code = NotFound desc = could not find container \"fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56\": container with ID starting with fd7b8f6f212fe686375579390ab70b1b61b31cbc5a70a20adf4248b269090d56 not found: ID does not exist" Nov 02 08:47:11.391870 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.391840 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:11.418636 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.418608 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7"] Nov 02 08:47:11.418844 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.418824 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" podUID="83079646-b7f8-4ccc-bac2-3e1b26fa630c" containerName="payload-pre-processing" containerID="cri-o://a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4" gracePeriod=30 Nov 02 08:47:11.559037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.558892 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:11.585774 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.585658 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq"] Nov 02 08:47:11.585991 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.585976 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d51175-8bb5bb8b9-6gtjq"] Nov 02 08:47:11.658120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.658087 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:47:11.683835 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.683805 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr"] Nov 02 08:47:11.690143 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.690111 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d51175-7f8f8f648-sq8sr"] Nov 02 08:47:11.700545 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.700524 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:47:11.715500 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.715472 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:11.719456 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.719437 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83079646-b7f8-4ccc-bac2-3e1b26fa630c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83079646-b7f8-4ccc-bac2-3e1b26fa630c-plugins-config-volume\") pod \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\" (UID: \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\") " Nov 02 08:47:11.719542 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.719474 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c0574a-cf84-4f48-9e56-15da5225e58d-kube-api-access-9kpkn\" (UniqueName: \"kubernetes.io/projected/c6c0574a-cf84-4f48-9e56-15da5225e58d-kube-api-access-9kpkn\") pod \"c6c0574a-cf84-4f48-9e56-15da5225e58d\" (UID: \"c6c0574a-cf84-4f48-9e56-15da5225e58d\") " Nov 02 08:47:11.719602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.719575 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83079646-b7f8-4ccc-bac2-3e1b26fa630c-kube-api-access-lck6z\" (UniqueName: \"kubernetes.io/projected/83079646-b7f8-4ccc-bac2-3e1b26fa630c-kube-api-access-lck6z\") pod \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\" (UID: \"83079646-b7f8-4ccc-bac2-3e1b26fa630c\") " Nov 02 08:47:11.719686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.719670 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c0574a-cf84-4f48-9e56-15da5225e58d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c0574a-cf84-4f48-9e56-15da5225e58d-plugins-config-volume\") pod \"c6c0574a-cf84-4f48-9e56-15da5225e58d\" (UID: \"c6c0574a-cf84-4f48-9e56-15da5225e58d\") " Nov 02 08:47:11.719989 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.719855 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83079646-b7f8-4ccc-bac2-3e1b26fa630c-plugins-config-volume" pod "83079646-b7f8-4ccc-bac2-3e1b26fa630c" (UID: "83079646-b7f8-4ccc-bac2-3e1b26fa630c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:47:11.720091 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.720016 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c0574a-cf84-4f48-9e56-15da5225e58d-plugins-config-volume" pod "c6c0574a-cf84-4f48-9e56-15da5225e58d" (UID: "c6c0574a-cf84-4f48-9e56-15da5225e58d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:47:11.721761 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.721738 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c0574a-cf84-4f48-9e56-15da5225e58d-kube-api-access-9kpkn" pod "c6c0574a-cf84-4f48-9e56-15da5225e58d" (UID: "c6c0574a-cf84-4f48-9e56-15da5225e58d"). InnerVolumeSpecName "kube-api-access-9kpkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:47:11.721839 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.721774 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83079646-b7f8-4ccc-bac2-3e1b26fa630c-kube-api-access-lck6z" pod "83079646-b7f8-4ccc-bac2-3e1b26fa630c" (UID: "83079646-b7f8-4ccc-bac2-3e1b26fa630c"). InnerVolumeSpecName "kube-api-access-lck6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:47:11.790970 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.790936 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf"] Nov 02 08:47:11.794913 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.794887 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d51175-787c6bc64c-2dglf"] Nov 02 08:47:11.821278 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.821254 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kpkn\" (UniqueName: \"kubernetes.io/projected/c6c0574a-cf84-4f48-9e56-15da5225e58d-kube-api-access-9kpkn\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.821278 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.821278 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lck6z\" (UniqueName: \"kubernetes.io/projected/83079646-b7f8-4ccc-bac2-3e1b26fa630c-kube-api-access-lck6z\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.821425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.821288 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c0574a-cf84-4f48-9e56-15da5225e58d-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.821425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.821297 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83079646-b7f8-4ccc-bac2-3e1b26fa630c-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:47:11.897224 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.897105 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b"] Nov 02 08:47:11.904062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:11.904018 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d91f6f-6f985bf95-4fw7b"] Nov 02 08:47:12.325617 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.325580 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:12.340272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.340238 6160 generic.go:356] "Generic (PLEG): container finished" podID="c6c0574a-cf84-4f48-9e56-15da5225e58d" containerID="2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da" exitCode=0 Nov 02 08:47:12.340399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.340288 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" Nov 02 08:47:12.340399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.340330 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" event={"ID":"c6c0574a-cf84-4f48-9e56-15da5225e58d","Type":"ContainerDied","Data":"2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da"} Nov 02 08:47:12.340399 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.340387 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk" event={"ID":"c6c0574a-cf84-4f48-9e56-15da5225e58d","Type":"ContainerDied","Data":"bb10d8fdbddb761125c8adc67f8dc9dfd86123d1a8491892cdaaf93325109349"} Nov 02 08:47:12.340555 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.340418 6160 scope.go:122] "RemoveContainer" containerID="2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da" Nov 02 08:47:12.342731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.342709 6160 generic.go:356] "Generic (PLEG): container finished" podID="83079646-b7f8-4ccc-bac2-3e1b26fa630c" containerID="a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4" exitCode=0 Nov 02 08:47:12.342859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.342749 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" Nov 02 08:47:12.342859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.342774 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" event={"ID":"83079646-b7f8-4ccc-bac2-3e1b26fa630c","Type":"ContainerDied","Data":"a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4"} Nov 02 08:47:12.342859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.342810 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7" event={"ID":"83079646-b7f8-4ccc-bac2-3e1b26fa630c","Type":"ContainerDied","Data":"cd6acf4091c2b612947c0af85c241d636ffc1a2ed66e20e04ce2cff1e62d7f06"} Nov 02 08:47:12.351529 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.351507 6160 scope.go:122] "RemoveContainer" containerID="2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da" Nov 02 08:47:12.351936 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:47:12.351793 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da\": container with ID starting with 2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da not found: ID does not exist" containerID="2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da" Nov 02 08:47:12.351936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.351817 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da"} err="failed to get container status \"2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da\": rpc error: code = NotFound desc = could not find container \"2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da\": container with ID starting with 2fbc9d21753c6d958c806f3208f1c1983fb7c2a3a508e5e6ca7aae01741969da not found: ID does not exist" Nov 02 08:47:12.351936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.351835 6160 scope.go:122] "RemoveContainer" containerID="a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4" Nov 02 08:47:12.360891 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.360873 6160 scope.go:122] "RemoveContainer" containerID="a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4" Nov 02 08:47:12.361166 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:47:12.361146 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4\": container with ID starting with a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4 not found: ID does not exist" containerID="a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4" Nov 02 08:47:12.361239 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.361189 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4"} err="failed to get container status \"a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4\": rpc error: code = NotFound desc = could not find container \"a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4\": container with ID starting with a3d01aec50332694864739f4ad1dddfee44e431b61213df52e25ffc4a621e8d4 not found: ID does not exist" Nov 02 08:47:12.492735 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.492657 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7"] Nov 02 08:47:12.498561 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.498533 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d91f6f-587b87dc9-4hkw7"] Nov 02 08:47:12.601492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.601455 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk"] Nov 02 08:47:12.607016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.606990 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d91f6f-7cff4575b-8mxjk"] Nov 02 08:47:12.704804 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.704771 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:12.994457 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:12.994425 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:13.164151 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:13.164114 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:13.583690 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:13.583656 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb4d9c6-5abb-4536-a36e-50390e6dabf0" path="/var/lib/kubelet/pods/5bb4d9c6-5abb-4536-a36e-50390e6dabf0/volumes" Nov 02 08:47:13.584140 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:13.584120 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83079646-b7f8-4ccc-bac2-3e1b26fa630c" path="/var/lib/kubelet/pods/83079646-b7f8-4ccc-bac2-3e1b26fa630c/volumes" Nov 02 08:47:13.584591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:13.584572 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b483ec-087c-4c10-973c-fea9a66a1073" path="/var/lib/kubelet/pods/96b483ec-087c-4c10-973c-fea9a66a1073/volumes" Nov 02 08:47:13.585340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:13.585320 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d93ffc-c954-4760-8606-29cf9767c36b" path="/var/lib/kubelet/pods/a9d93ffc-c954-4760-8606-29cf9767c36b/volumes" Nov 02 08:47:13.585741 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:13.585723 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c0574a-cf84-4f48-9e56-15da5225e58d" path="/var/lib/kubelet/pods/c6c0574a-cf84-4f48-9e56-15da5225e58d/volumes" Nov 02 08:47:13.586141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:13.586124 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c821cfa2-aa32-4fa8-a9b6-f52884cd658a" path="/var/lib/kubelet/pods/c821cfa2-aa32-4fa8-a9b6-f52884cd658a/volumes" Nov 02 08:47:13.610310 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:13.610279 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:17.329285 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:17.329247 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:18.471961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:18.471926 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:19.314601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:19.314565 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:19.495256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:19.495219 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:19.664636 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:19.664602 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:21.151295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.151253 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:21.616256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616224 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs"] Nov 02 08:47:21.616635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616622 6160 cpu_manager.go:414] "containerMap: removing container" podUID="b20d3925-caaa-46ae-a508-808eb42aac2e" containerName="payload-pre-processing" Nov 02 08:47:21.616686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616636 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20d3925-caaa-46ae-a508-808eb42aac2e" containerName="payload-pre-processing" Nov 02 08:47:21.616686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616645 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c821cfa2-aa32-4fa8-a9b6-f52884cd658a" containerName="maas-api" Nov 02 08:47:21.616686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616651 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c821cfa2-aa32-4fa8-a9b6-f52884cd658a" containerName="maas-api" Nov 02 08:47:21.616686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616660 6160 cpu_manager.go:414] "containerMap: removing container" podUID="5bb4d9c6-5abb-4536-a36e-50390e6dabf0" containerName="payload-pre-processing" Nov 02 08:47:21.616686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616665 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb4d9c6-5abb-4536-a36e-50390e6dabf0" containerName="payload-pre-processing" Nov 02 08:47:21.616686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616674 6160 cpu_manager.go:414] "containerMap: removing container" podUID="96b483ec-087c-4c10-973c-fea9a66a1073" containerName="maas-api" Nov 02 08:47:21.616686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616678 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b483ec-087c-4c10-973c-fea9a66a1073" containerName="maas-api" Nov 02 08:47:21.616686 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616686 6160 cpu_manager.go:414] "containerMap: removing container" podUID="83079646-b7f8-4ccc-bac2-3e1b26fa630c" containerName="payload-pre-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616691 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83079646-b7f8-4ccc-bac2-3e1b26fa630c" containerName="payload-pre-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616700 6160 cpu_manager.go:414] "containerMap: removing container" podUID="d81bdb8e-3bce-4589-b74e-29be4446dd50" containerName="maas-api" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616704 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81bdb8e-3bce-4589-b74e-29be4446dd50" containerName="maas-api" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616713 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c6c0574a-cf84-4f48-9e56-15da5225e58d" containerName="payload-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616717 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c0574a-cf84-4f48-9e56-15da5225e58d" containerName="payload-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616727 6160 cpu_manager.go:414] "containerMap: removing container" podUID="9b8d2324-5246-4559-81d8-507723e7ab6b" containerName="maas-api" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616731 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8d2324-5246-4559-81d8-507723e7ab6b" containerName="maas-api" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616740 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a9d93ffc-c954-4760-8606-29cf9767c36b" containerName="payload-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616744 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d93ffc-c954-4760-8606-29cf9767c36b" containerName="payload-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616800 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c0574a-cf84-4f48-9e56-15da5225e58d" containerName="payload-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616809 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8d2324-5246-4559-81d8-507723e7ab6b" containerName="maas-api" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616816 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d93ffc-c954-4760-8606-29cf9767c36b" containerName="payload-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616824 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20d3925-caaa-46ae-a508-808eb42aac2e" containerName="payload-pre-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616829 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c821cfa2-aa32-4fa8-a9b6-f52884cd658a" containerName="maas-api" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616834 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb4d9c6-5abb-4536-a36e-50390e6dabf0" containerName="payload-pre-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616838 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b483ec-087c-4c10-973c-fea9a66a1073" containerName="maas-api" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616843 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="83079646-b7f8-4ccc-bac2-3e1b26fa630c" containerName="payload-pre-processing" Nov 02 08:47:21.616918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.616850 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81bdb8e-3bce-4589-b74e-29be4446dd50" containerName="maas-api" Nov 02 08:47:21.621635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.621618 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.671901 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.671305 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-8f1e34-openshift-default-dockercfg-pgxdk\"" Nov 02 08:47:21.719816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.719779 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs"] Nov 02 08:47:21.723573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.723525 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmdg\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-kube-api-access-rhmdg\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.723789 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.723764 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-istio-token\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.723909 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.723806 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-certs\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.723965 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.723939 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-credential-socket\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.724004 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.723972 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-data\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.724004 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.724000 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-envoy\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.724168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.724142 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95021cec-10e1-4c10-b122-999e183abde6-istiod-ca-cert\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.724263 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.724202 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95021cec-10e1-4c10-b122-999e183abde6-istio-podinfo\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.724263 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.724245 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-socket\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.743708 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.743663 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:21.825379 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825337 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmdg\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-kube-api-access-rhmdg\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.825649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825382 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-istio-token\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.825649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825422 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-certs\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.825649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825507 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-credential-socket\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.825649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825540 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-data\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.825649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825583 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-envoy\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.825649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825628 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95021cec-10e1-4c10-b122-999e183abde6-istiod-ca-cert\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.825649 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825650 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95021cec-10e1-4c10-b122-999e183abde6-istio-podinfo\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.826042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825694 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-socket\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.826042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.825976 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-data\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.826042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.826017 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-credential-socket\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.826264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.826063 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-certs\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.826264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.826252 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-socket\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.826401 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.826381 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95021cec-10e1-4c10-b122-999e183abde6-istiod-ca-cert\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.828037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.828017 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-envoy\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.828304 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.828286 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95021cec-10e1-4c10-b122-999e183abde6-istio-podinfo\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.878708 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.878631 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-istio-token\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.878850 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.878701 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmdg\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-kube-api-access-rhmdg\") pod \"e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.934818 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.934767 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:21.986880 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:21.986842 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:22.362042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:22.362018 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs"] Nov 02 08:47:22.364134 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:47:22.364105 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95021cec_10e1_4c10_b122_999e183abde6.slice/crio-4ee710901f1827d873c385beeb967150d29f01b1c4724deead19fa3fe529ea9e WatchSource:0}: Error finding container 4ee710901f1827d873c385beeb967150d29f01b1c4724deead19fa3fe529ea9e: Status 404 returned error can't find the container with id 4ee710901f1827d873c385beeb967150d29f01b1c4724deead19fa3fe529ea9e Nov 02 08:47:22.366460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:22.366419 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:47:22.366592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:22.366499 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:47:22.366592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:22.366543 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:47:22.405892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:22.405864 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" event={"ID":"95021cec-10e1-4c10-b122-999e183abde6","Type":"ContainerStarted","Data":"4ee710901f1827d873c385beeb967150d29f01b1c4724deead19fa3fe529ea9e"} Nov 02 08:47:23.413936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:23.413893 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" event={"ID":"95021cec-10e1-4c10-b122-999e183abde6","Type":"ContainerStarted","Data":"a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff"} Nov 02 08:47:23.572106 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:23.572031 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" podStartSLOduration=2.572012563 podStartE2EDuration="2.572012563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:47:23.569835085 +0000 UTC m=+2442.123339283" watchObservedRunningTime="2026-11-02 08:47:23.572012563 +0000 UTC m=+2442.125516762" Nov 02 08:47:23.935747 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:23.935713 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:23.940643 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:23.940617 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:24.426079 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:24.422365 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:24.427048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:24.427024 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:47:24.978186 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:24.978118 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:26.257290 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:26.257247 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:26.492454 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:26.492415 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:26.657766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:26.657677 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:29.513542 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:29.513508 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:29.696860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:29.696815 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:30.022084 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.022045 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g"] Nov 02 08:47:30.029931 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.029898 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:30.083950 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.083915 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-8f1e34-dockercfg-d8mgs\"" Nov 02 08:47:30.090362 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.090328 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g"] Nov 02 08:47:30.120922 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.120892 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-8f1e34\"" Nov 02 08:47:30.129433 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.129409 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g\" (UID: \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:30.129526 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.129459 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqm6\" (UniqueName: \"kubernetes.io/projected/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-kube-api-access-zzqm6\") pod \"payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g\" (UID: \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:30.143312 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.143278 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:30.204610 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.204570 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp"] Nov 02 08:47:30.208403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.208386 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:30.230713 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.230678 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g\" (UID: \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:30.230915 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.230727 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqm6\" (UniqueName: \"kubernetes.io/projected/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-kube-api-access-zzqm6\") pod \"payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g\" (UID: \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:30.231391 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.231368 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g\" (UID: \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:30.262396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.262358 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp"] Nov 02 08:47:30.284394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.284303 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqm6\" (UniqueName: \"kubernetes.io/projected/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-kube-api-access-zzqm6\") pod \"payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g\" (UID: \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:30.331859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.331818 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xvds\" (UniqueName: \"kubernetes.io/projected/661e4ae8-b1e8-4985-8720-1d9401634a88-kube-api-access-2xvds\") pod \"payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp\" (UID: \"661e4ae8-b1e8-4985-8720-1d9401634a88\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:30.331859 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.331861 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661e4ae8-b1e8-4985-8720-1d9401634a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp\" (UID: \"661e4ae8-b1e8-4985-8720-1d9401634a88\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:30.356740 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.356697 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:30.433105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.433068 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xvds\" (UniqueName: \"kubernetes.io/projected/661e4ae8-b1e8-4985-8720-1d9401634a88-kube-api-access-2xvds\") pod \"payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp\" (UID: \"661e4ae8-b1e8-4985-8720-1d9401634a88\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:30.433262 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.433114 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661e4ae8-b1e8-4985-8720-1d9401634a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp\" (UID: \"661e4ae8-b1e8-4985-8720-1d9401634a88\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:30.433986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.433965 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661e4ae8-b1e8-4985-8720-1d9401634a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp\" (UID: \"661e4ae8-b1e8-4985-8720-1d9401634a88\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:30.486772 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.486740 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xvds\" (UniqueName: \"kubernetes.io/projected/661e4ae8-b1e8-4985-8720-1d9401634a88-kube-api-access-2xvds\") pod \"payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp\" (UID: \"661e4ae8-b1e8-4985-8720-1d9401634a88\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:30.521693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.521664 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:30.730232 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.730200 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g"] Nov 02 08:47:30.731772 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:47:30.731746 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18eabf3d_e6cb_41d9_9a0e_f3ce8b6e853e.slice/crio-9e54bf78451ae73e8d680d9494f18af049eff9eb1e1e851d30a50890a94f16eb WatchSource:0}: Error finding container 9e54bf78451ae73e8d680d9494f18af049eff9eb1e1e851d30a50890a94f16eb: Status 404 returned error can't find the container with id 9e54bf78451ae73e8d680d9494f18af049eff9eb1e1e851d30a50890a94f16eb Nov 02 08:47:30.890849 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:30.890823 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp"] Nov 02 08:47:30.892987 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:47:30.892959 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661e4ae8_b1e8_4985_8720_1d9401634a88.slice/crio-bfe7ef5ec95770ef09f41cd9cc1216fcdd1de69f84ec26fac35921f998a9c4d2 WatchSource:0}: Error finding container bfe7ef5ec95770ef09f41cd9cc1216fcdd1de69f84ec26fac35921f998a9c4d2: Status 404 returned error can't find the container with id bfe7ef5ec95770ef09f41cd9cc1216fcdd1de69f84ec26fac35921f998a9c4d2 Nov 02 08:47:31.463441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:31.463398 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" event={"ID":"661e4ae8-b1e8-4985-8720-1d9401634a88","Type":"ContainerStarted","Data":"8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844"} Nov 02 08:47:31.463657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:31.463461 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" event={"ID":"661e4ae8-b1e8-4985-8720-1d9401634a88","Type":"ContainerStarted","Data":"bfe7ef5ec95770ef09f41cd9cc1216fcdd1de69f84ec26fac35921f998a9c4d2"} Nov 02 08:47:31.463657 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:31.463491 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:31.465046 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:31.465008 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" event={"ID":"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e","Type":"ContainerStarted","Data":"3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478"} Nov 02 08:47:31.465193 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:31.465059 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" event={"ID":"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e","Type":"ContainerStarted","Data":"9e54bf78451ae73e8d680d9494f18af049eff9eb1e1e851d30a50890a94f16eb"} Nov 02 08:47:31.465193 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:31.465096 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:31.569789 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:31.569735 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" podStartSLOduration=1.569720797 podStartE2EDuration="1.569720797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:47:31.569671723 +0000 UTC m=+2450.123175932" watchObservedRunningTime="2026-11-02 08:47:31.569720797 +0000 UTC m=+2450.123224995" Nov 02 08:47:31.677498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:31.677434 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" podStartSLOduration=2.677421255 podStartE2EDuration="2.677421255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:47:31.676030723 +0000 UTC m=+2450.229534920" watchObservedRunningTime="2026-11-02 08:47:31.677421255 +0000 UTC m=+2450.230925446" Nov 02 08:47:34.928164 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:34.928127 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:39.922083 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:39.922039 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:40.521166 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:47:40.521128 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:47:42.473904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:42.473873 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:47:42.474558 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:42.473957 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:47:42.598135 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:42.597511 6160 scope.go:122] "RemoveContainer" containerID="5e6abf256f2fcd48d55e89156cf851e8fcacf6bd02d3ae755a34cc0c8bb3b2ce" Nov 02 08:47:46.059151 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.059113 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b"] Nov 02 08:47:46.064565 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.064545 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.105069 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.105031 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47149d9-ef03-4b19-ae24-7215cdd1d93a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.105264 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.105092 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.105313 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.105256 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljtd\" (UniqueName: \"kubernetes.io/projected/a47149d9-ef03-4b19-ae24-7215cdd1d93a-kube-api-access-sljtd\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.105358 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.105322 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-metrics-certs\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.124668 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.124636 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8f1e34-serving-cert\"" Nov 02 08:47:46.170499 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.170460 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b"] Nov 02 08:47:46.206245 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.206207 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljtd\" (UniqueName: \"kubernetes.io/projected/a47149d9-ef03-4b19-ae24-7215cdd1d93a-kube-api-access-sljtd\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.206442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.206273 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-metrics-certs\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.206442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.206329 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47149d9-ef03-4b19-ae24-7215cdd1d93a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.206442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.206378 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.206631 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:47:46.206501 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-serving-cert: secret "maas-api-e2e-praxis-a-8f1e34-serving-cert" not found Nov 02 08:47:46.206631 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:47:46.206561 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls podName:a47149d9-ef03-4b19-ae24-7215cdd1d93a nodeName:}" failed. No retries permitted until 2026-11-02 08:47:46.7065449 +0000 UTC m=+2465.260049076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls") pod "maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" (UID: "a47149d9-ef03-4b19-ae24-7215cdd1d93a") : secret "maas-api-e2e-praxis-a-8f1e34-serving-cert" not found Nov 02 08:47:46.207055 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.207034 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47149d9-ef03-4b19-ae24-7215cdd1d93a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.209078 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.209054 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-metrics-certs\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.260976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.260932 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljtd\" (UniqueName: \"kubernetes.io/projected/a47149d9-ef03-4b19-ae24-7215cdd1d93a-kube-api-access-sljtd\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.321874 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.321799 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:46.482492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.482448 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:46.644643 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.644543 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:46.712443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.712405 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.715098 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.715074 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:46.978386 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:46.978355 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:47.347550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:47.347521 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b"] Nov 02 08:47:47.350098 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:47:47.350074 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47149d9_ef03_4b19_ae24_7215cdd1d93a.slice/crio-9a1bbd35c9ffccade78f90fa6149a98ccc6a1f625b90b67ef86c5e8cb8093bde WatchSource:0}: Error finding container 9a1bbd35c9ffccade78f90fa6149a98ccc6a1f625b90b67ef86c5e8cb8093bde: Status 404 returned error can't find the container with id 9a1bbd35c9ffccade78f90fa6149a98ccc6a1f625b90b67ef86c5e8cb8093bde Nov 02 08:47:47.351946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:47.351929 6160 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:47:47.558840 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:47.558798 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" event={"ID":"a47149d9-ef03-4b19-ae24-7215cdd1d93a","Type":"ContainerStarted","Data":"9a1bbd35c9ffccade78f90fa6149a98ccc6a1f625b90b67ef86c5e8cb8093bde"} Nov 02 08:47:48.566452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:48.566416 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" event={"ID":"a47149d9-ef03-4b19-ae24-7215cdd1d93a","Type":"ContainerStarted","Data":"d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3"} Nov 02 08:47:48.566871 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:48.566571 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:48.676122 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:48.676072 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" podStartSLOduration=1.663145563 podStartE2EDuration="2.676058504s" totalImagesPullingTime="1.012912941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:47:48.673447589 +0000 UTC m=+2467.226951777" watchObservedRunningTime="2026-11-02 08:47:48.676058504 +0000 UTC m=+2467.229562679" Nov 02 08:47:51.294616 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:51.294574 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:53.681270 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:53.681236 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:54.461918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:54.461883 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:54.578280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:54.578241 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:47:56.151972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:47:56.151936 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:00.055719 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:00.055684 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:00.232577 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:00.232542 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:00.390248 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:00.390138 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:01.001830 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:01.001790 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:02.950351 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:02.950317 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:03.631904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:03.631864 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:06.346532 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:06.346480 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:07.690428 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:07.690397 6160 generic.go:356] "Generic (PLEG): container finished" podID="95021cec-10e1-4c10-b122-999e183abde6" containerID="a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff" exitCode=0 Nov 02 08:48:07.690872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:07.690475 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" event={"ID":"95021cec-10e1-4c10-b122-999e183abde6","Type":"ContainerDied","Data":"a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff"} Nov 02 08:48:07.690872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:07.690848 6160 scope.go:122] "RemoveContainer" containerID="a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff" Nov 02 08:48:07.692088 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:07.692057 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:48:07.692221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:07.692130 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:48:07.692221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:07.692160 6160 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 08:48:08.698511 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:08.698475 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" event={"ID":"95021cec-10e1-4c10-b122-999e183abde6","Type":"ContainerStarted","Data":"0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d"} Nov 02 08:48:08.935537 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:08.935500 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:48:08.936707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:08.936684 6160 patch_prober.go:29] interesting pod/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 08:48:08.936816 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:08.936747 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 08:48:09.428332 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:09.428291 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:48:09.936113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:09.936077 6160 patch_prober.go:29] interesting pod/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 08:48:09.936529 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:09.936128 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 08:48:10.541788 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:48:10.541748 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:48:10.585031 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:10.584992 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:10.935308 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:10.935278 6160 patch_prober.go:29] interesting pod/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 08:48:10.935480 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:10.935342 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 08:48:11.935229 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:11.935161 6160 patch_prober.go:29] interesting pod/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 08:48:11.935635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:11.935241 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 08:48:12.832784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:12.832729 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:12.935477 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:12.935436 6160 patch_prober.go:29] interesting pod/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 08:48:12.935879 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:12.935505 6160 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 08:48:13.939340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:13.939308 6160 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:48:13.940287 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:13.940267 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:48:17.662065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:17.662023 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:19.568426 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:19.568393 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb"] Nov 02 08:48:19.568884 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:19.568594 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" containerID="cri-o://fbab1e84bb3104efdea65a303a0541159ce5d24436f5d032235734fede1d7810" gracePeriod=30 Nov 02 08:48:19.615755 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:19.615724 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd"] Nov 02 08:48:19.615961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:19.615939 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" containerID="cri-o://f5dfc12cdf16ca735e2a0b2e1fc7623bdcf6ea3b577c7fd66a28132d83720e44" gracePeriod=30 Nov 02 08:48:19.618075 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:19.618045 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds"] Nov 02 08:48:19.618285 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:19.618262 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" containerID="cri-o://9786f4c8cf0c5eb45703ffa571e30e054aeb6fe1b7ba3b974f384e60817c8bd9" gracePeriod=30 Nov 02 08:48:19.661944 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:19.661911 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p"] Nov 02 08:48:19.662118 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:19.662098 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" containerID="cri-o://2593ff2802b4db422d2a579b6cf51d278acfc8a63bd7fb5d9a3659bffa9c36f9" gracePeriod=30 Nov 02 08:48:20.742352 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:20.742312 6160 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l54fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:48:20.742736 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:20.742373 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:48:22.030715 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:22.030677 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:22.824311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:22.824277 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:23.290540 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:23.290496 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:24.182112 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.182055 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:24.514491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.514406 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:24.821214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.819645 6160 generic.go:356] "Generic (PLEG): container finished" podID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerID="f5dfc12cdf16ca735e2a0b2e1fc7623bdcf6ea3b577c7fd66a28132d83720e44" exitCode=0 Nov 02 08:48:24.821214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.819824 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" event={"ID":"c724ad29-9eb6-4e0c-9ff3-0f949e462de0","Type":"ContainerDied","Data":"f5dfc12cdf16ca735e2a0b2e1fc7623bdcf6ea3b577c7fd66a28132d83720e44"} Nov 02 08:48:24.822579 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.822535 6160 generic.go:356] "Generic (PLEG): container finished" podID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerID="fbab1e84bb3104efdea65a303a0541159ce5d24436f5d032235734fede1d7810" exitCode=0 Nov 02 08:48:24.822860 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.822838 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" event={"ID":"20d6fe81-2539-40b8-b08c-8ed5033da34c","Type":"ContainerDied","Data":"fbab1e84bb3104efdea65a303a0541159ce5d24436f5d032235734fede1d7810"} Nov 02 08:48:24.825322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.825295 6160 generic.go:356] "Generic (PLEG): container finished" podID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerID="9786f4c8cf0c5eb45703ffa571e30e054aeb6fe1b7ba3b974f384e60817c8bd9" exitCode=0 Nov 02 08:48:24.825630 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.825609 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" event={"ID":"ec99e3d7-5c3d-4596-9b34-bdecbd94643f","Type":"ContainerDied","Data":"9786f4c8cf0c5eb45703ffa571e30e054aeb6fe1b7ba3b974f384e60817c8bd9"} Nov 02 08:48:24.827925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.827822 6160 generic.go:356] "Generic (PLEG): container finished" podID="73012162-12a0-467c-b849-1d9dd494c0b7" containerID="2593ff2802b4db422d2a579b6cf51d278acfc8a63bd7fb5d9a3659bffa9c36f9" exitCode=0 Nov 02 08:48:24.827925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.827858 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" event={"ID":"73012162-12a0-467c-b849-1d9dd494c0b7","Type":"ContainerDied","Data":"2593ff2802b4db422d2a579b6cf51d278acfc8a63bd7fb5d9a3659bffa9c36f9"} Nov 02 08:48:24.893145 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.893120 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:48:24.937493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.937461 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:48:24.951960 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.951925 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-credential-socket\") pod \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " Nov 02 08:48:24.952210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.951978 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-certs\") pod \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " Nov 02 08:48:24.952210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952003 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d6fe81-2539-40b8-b08c-8ed5033da34c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6fe81-2539-40b8-b08c-8ed5033da34c-istiod-ca-cert\") pod \"20d6fe81-2539-40b8-b08c-8ed5033da34c\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " Nov 02 08:48:24.952210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952034 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istiod-ca-cert\") pod \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " Nov 02 08:48:24.952210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952061 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-credential-socket\") pod \"20d6fe81-2539-40b8-b08c-8ed5033da34c\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " Nov 02 08:48:24.952210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952150 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-podinfo\") pod \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " Nov 02 08:48:24.952210 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952198 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-token\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-token\") pod \"20d6fe81-2539-40b8-b08c-8ed5033da34c\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952251 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-token\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-token\") pod \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952275 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-kube-api-access-lh9xb\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-kube-api-access-lh9xb\") pod \"20d6fe81-2539-40b8-b08c-8ed5033da34c\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952300 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-socket\") pod \"20d6fe81-2539-40b8-b08c-8ed5033da34c\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952319 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-data\") pod \"20d6fe81-2539-40b8-b08c-8ed5033da34c\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952341 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-envoy\") pod \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952369 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-podinfo\") pod \"20d6fe81-2539-40b8-b08c-8ed5033da34c\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952388 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-certs\") pod \"20d6fe81-2539-40b8-b08c-8ed5033da34c\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952409 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-envoy\") pod \"20d6fe81-2539-40b8-b08c-8ed5033da34c\" (UID: \"20d6fe81-2539-40b8-b08c-8ed5033da34c\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952437 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-data\") pod \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952480 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-socket\") pod \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952477 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d6fe81-2539-40b8-b08c-8ed5033da34c-istiod-ca-cert" pod "20d6fe81-2539-40b8-b08c-8ed5033da34c" (UID: "20d6fe81-2539-40b8-b08c-8ed5033da34c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952506 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-kube-api-access-bjts7\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-kube-api-access-bjts7\") pod \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\" (UID: \"c724ad29-9eb6-4e0c-9ff3-0f949e462de0\") " Nov 02 08:48:24.952726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952555 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-credential-socket" pod "c724ad29-9eb6-4e0c-9ff3-0f949e462de0" (UID: "c724ad29-9eb6-4e0c-9ff3-0f949e462de0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.953766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952779 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-credential-socket" pod "20d6fe81-2539-40b8-b08c-8ed5033da34c" (UID: "20d6fe81-2539-40b8-b08c-8ed5033da34c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.953766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952953 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:24.953766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952974 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6fe81-2539-40b8-b08c-8ed5033da34c-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:24.953766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.952990 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:24.953766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.953079 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istiod-ca-cert" pod "c724ad29-9eb6-4e0c-9ff3-0f949e462de0" (UID: "c724ad29-9eb6-4e0c-9ff3-0f949e462de0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:48:24.953766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.953417 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-socket" pod "20d6fe81-2539-40b8-b08c-8ed5033da34c" (UID: "20d6fe81-2539-40b8-b08c-8ed5033da34c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.953766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.953673 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-data" pod "c724ad29-9eb6-4e0c-9ff3-0f949e462de0" (UID: "c724ad29-9eb6-4e0c-9ff3-0f949e462de0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.954250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.953855 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-socket" pod "c724ad29-9eb6-4e0c-9ff3-0f949e462de0" (UID: "c724ad29-9eb6-4e0c-9ff3-0f949e462de0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.954250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.954009 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-certs" pod "c724ad29-9eb6-4e0c-9ff3-0f949e462de0" (UID: "c724ad29-9eb6-4e0c-9ff3-0f949e462de0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.954250 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.954229 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-data" pod "20d6fe81-2539-40b8-b08c-8ed5033da34c" (UID: "20d6fe81-2539-40b8-b08c-8ed5033da34c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.954452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.954406 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-certs" pod "20d6fe81-2539-40b8-b08c-8ed5033da34c" (UID: "20d6fe81-2539-40b8-b08c-8ed5033da34c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.958081 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.958057 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-podinfo" pod "c724ad29-9eb6-4e0c-9ff3-0f949e462de0" (UID: "c724ad29-9eb6-4e0c-9ff3-0f949e462de0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:48:24.959375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.959348 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-podinfo" pod "20d6fe81-2539-40b8-b08c-8ed5033da34c" (UID: "20d6fe81-2539-40b8-b08c-8ed5033da34c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:48:24.959486 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.959384 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-kube-api-access-lh9xb" pod "20d6fe81-2539-40b8-b08c-8ed5033da34c" (UID: "20d6fe81-2539-40b8-b08c-8ed5033da34c"). InnerVolumeSpecName "kube-api-access-lh9xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:24.959778 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.959757 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-kube-api-access-bjts7" pod "c724ad29-9eb6-4e0c-9ff3-0f949e462de0" (UID: "c724ad29-9eb6-4e0c-9ff3-0f949e462de0"). InnerVolumeSpecName "kube-api-access-bjts7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:24.959967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.959953 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-token" pod "c724ad29-9eb6-4e0c-9ff3-0f949e462de0" (UID: "c724ad29-9eb6-4e0c-9ff3-0f949e462de0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:24.961234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.961147 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-envoy" pod "c724ad29-9eb6-4e0c-9ff3-0f949e462de0" (UID: "c724ad29-9eb6-4e0c-9ff3-0f949e462de0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.961322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.961225 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-token" pod "20d6fe81-2539-40b8-b08c-8ed5033da34c" (UID: "20d6fe81-2539-40b8-b08c-8ed5033da34c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:24.961493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.961473 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-envoy" pod "20d6fe81-2539-40b8-b08c-8ed5033da34c" (UID: "20d6fe81-2539-40b8-b08c-8ed5033da34c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:24.983327 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.983295 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:48:24.992311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:24.992274 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:48:25.053666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053571 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-socket\") pod \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " Nov 02 08:48:25.053666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053617 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-credential-socket\") pod \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " Nov 02 08:48:25.053666 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053640 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-credential-socket\") pod \"73012162-12a0-467c-b849-1d9dd494c0b7\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " Nov 02 08:48:25.053969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053695 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73012162-12a0-467c-b849-1d9dd494c0b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73012162-12a0-467c-b849-1d9dd494c0b7-istio-podinfo\") pod \"73012162-12a0-467c-b849-1d9dd494c0b7\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " Nov 02 08:48:25.053969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053761 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-certs\") pod \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " Nov 02 08:48:25.053969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053788 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-podinfo\") pod \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " Nov 02 08:48:25.053969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053869 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-envoy\") pod \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " Nov 02 08:48:25.053969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053896 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-socket" pod "ec99e3d7-5c3d-4596-9b34-bdecbd94643f" (UID: "ec99e3d7-5c3d-4596-9b34-bdecbd94643f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.053969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053916 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-certs\") pod \"73012162-12a0-467c-b849-1d9dd494c0b7\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " Nov 02 08:48:25.053969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053948 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-credential-socket" pod "ec99e3d7-5c3d-4596-9b34-bdecbd94643f" (UID: "ec99e3d7-5c3d-4596-9b34-bdecbd94643f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.053969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.053959 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-istio-token\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-istio-token\") pod \"73012162-12a0-467c-b849-1d9dd494c0b7\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " Nov 02 08:48:25.054350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054012 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-kube-api-access-r5l2v\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-kube-api-access-r5l2v\") pod \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " Nov 02 08:48:25.054350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054018 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-credential-socket" pod "73012162-12a0-467c-b849-1d9dd494c0b7" (UID: "73012162-12a0-467c-b849-1d9dd494c0b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.054350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054057 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-envoy\") pod \"73012162-12a0-467c-b849-1d9dd494c0b7\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " Nov 02 08:48:25.054350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054089 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-data\") pod \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " Nov 02 08:48:25.054350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054145 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73012162-12a0-467c-b849-1d9dd494c0b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73012162-12a0-467c-b849-1d9dd494c0b7-istiod-ca-cert\") pod \"73012162-12a0-467c-b849-1d9dd494c0b7\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " Nov 02 08:48:25.054350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054188 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-certs" pod "ec99e3d7-5c3d-4596-9b34-bdecbd94643f" (UID: "ec99e3d7-5c3d-4596-9b34-bdecbd94643f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.054350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054219 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istiod-ca-cert\") pod \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " Nov 02 08:48:25.054350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054247 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-socket\") pod \"73012162-12a0-467c-b849-1d9dd494c0b7\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " Nov 02 08:48:25.054350 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054292 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-token\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-token\") pod \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\" (UID: \"ec99e3d7-5c3d-4596-9b34-bdecbd94643f\") " Nov 02 08:48:25.054817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054367 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-kube-api-access-9hjph\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-kube-api-access-9hjph\") pod \"73012162-12a0-467c-b849-1d9dd494c0b7\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " Nov 02 08:48:25.054817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054402 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-data\") pod \"73012162-12a0-467c-b849-1d9dd494c0b7\" (UID: \"73012162-12a0-467c-b849-1d9dd494c0b7\") " Nov 02 08:48:25.054817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054476 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-data" pod "ec99e3d7-5c3d-4596-9b34-bdecbd94643f" (UID: "ec99e3d7-5c3d-4596-9b34-bdecbd94643f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.054817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054550 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73012162-12a0-467c-b849-1d9dd494c0b7-istiod-ca-cert" pod "73012162-12a0-467c-b849-1d9dd494c0b7" (UID: "73012162-12a0-467c-b849-1d9dd494c0b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:48:25.054817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.054680 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-certs" pod "73012162-12a0-467c-b849-1d9dd494c0b7" (UID: "73012162-12a0-467c-b849-1d9dd494c0b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.055095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.055045 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.055095 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.055082 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh9xb\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-kube-api-access-lh9xb\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.055236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.055097 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.055236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.055112 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.055236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.055127 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.055236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.055142 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056415 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056382 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-data" pod "73012162-12a0-467c-b849-1d9dd494c0b7" (UID: "73012162-12a0-467c-b849-1d9dd494c0b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.056609 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056586 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-socket" pod "73012162-12a0-467c-b849-1d9dd494c0b7" (UID: "73012162-12a0-467c-b849-1d9dd494c0b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.056702 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056680 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056705 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056728 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056743 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056756 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056768 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056782 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056795 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjts7\" (UniqueName: \"kubernetes.io/projected/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-kube-api-access-bjts7\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056808 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056823 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056837 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056852 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056865 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056879 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73012162-12a0-467c-b849-1d9dd494c0b7-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056892 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ad29-9eb6-4e0c-9ff3-0f949e462de0-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.056999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.056904 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d6fe81-2539-40b8-b08c-8ed5033da34c-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.057664 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.057326 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istiod-ca-cert" pod "ec99e3d7-5c3d-4596-9b34-bdecbd94643f" (UID: "ec99e3d7-5c3d-4596-9b34-bdecbd94643f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:48:25.058102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.058074 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-envoy" pod "ec99e3d7-5c3d-4596-9b34-bdecbd94643f" (UID: "ec99e3d7-5c3d-4596-9b34-bdecbd94643f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.058276 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.058250 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-envoy" pod "73012162-12a0-467c-b849-1d9dd494c0b7" (UID: "73012162-12a0-467c-b849-1d9dd494c0b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:48:25.058720 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.058694 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73012162-12a0-467c-b849-1d9dd494c0b7-istio-podinfo" pod "73012162-12a0-467c-b849-1d9dd494c0b7" (UID: "73012162-12a0-467c-b849-1d9dd494c0b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:48:25.059473 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.059442 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-podinfo" pod "ec99e3d7-5c3d-4596-9b34-bdecbd94643f" (UID: "ec99e3d7-5c3d-4596-9b34-bdecbd94643f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:48:25.059851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.059818 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-istio-token" pod "73012162-12a0-467c-b849-1d9dd494c0b7" (UID: "73012162-12a0-467c-b849-1d9dd494c0b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:25.060907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.060871 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-kube-api-access-r5l2v" pod "ec99e3d7-5c3d-4596-9b34-bdecbd94643f" (UID: "ec99e3d7-5c3d-4596-9b34-bdecbd94643f"). InnerVolumeSpecName "kube-api-access-r5l2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:25.061123 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.061095 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-kube-api-access-9hjph" pod "73012162-12a0-467c-b849-1d9dd494c0b7" (UID: "73012162-12a0-467c-b849-1d9dd494c0b7"). InnerVolumeSpecName "kube-api-access-9hjph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:25.061354 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.061331 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-token" pod "ec99e3d7-5c3d-4596-9b34-bdecbd94643f" (UID: "ec99e3d7-5c3d-4596-9b34-bdecbd94643f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:25.158425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158392 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158421 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158431 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158440 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hjph\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-kube-api-access-9hjph\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158452 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158467 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73012162-12a0-467c-b849-1d9dd494c0b7-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158475 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158487 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158496 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73012162-12a0-467c-b849-1d9dd494c0b7-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158505 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5l2v\" (UniqueName: \"kubernetes.io/projected/ec99e3d7-5c3d-4596-9b34-bdecbd94643f-kube-api-access-r5l2v\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.158661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.158513 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73012162-12a0-467c-b849-1d9dd494c0b7-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:48:25.834866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.834828 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" Nov 02 08:48:25.834866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.834854 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p" event={"ID":"73012162-12a0-467c-b849-1d9dd494c0b7","Type":"ContainerDied","Data":"59e491a6252ef7cbd7bd4c1cf0a7c0e731f60ab5c2d8036e40de2336f7cbf6e6"} Nov 02 08:48:25.835419 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.834917 6160 scope.go:122] "RemoveContainer" containerID="2593ff2802b4db422d2a579b6cf51d278acfc8a63bd7fb5d9a3659bffa9c36f9" Nov 02 08:48:25.836382 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.836354 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" event={"ID":"c724ad29-9eb6-4e0c-9ff3-0f949e462de0","Type":"ContainerDied","Data":"9e8db27a35e2b19acd3e8f16cb9b8d852754b2b9b288369f892b7d9cb86a927b"} Nov 02 08:48:25.836505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.836363 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd" Nov 02 08:48:25.837883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.837867 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" Nov 02 08:48:25.837883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.837878 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb" event={"ID":"20d6fe81-2539-40b8-b08c-8ed5033da34c","Type":"ContainerDied","Data":"5cf156bb24f351186a112420dd64059df92505d8901bf350f8b1dd7383c2e2b7"} Nov 02 08:48:25.839495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.839474 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" Nov 02 08:48:25.839588 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.839477 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds" event={"ID":"ec99e3d7-5c3d-4596-9b34-bdecbd94643f","Type":"ContainerDied","Data":"1c843b9b4c0c959a2119bc8d2e4e10f274a9ed9450fd4bda957d10f6dae51485"} Nov 02 08:48:25.846788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.846771 6160 scope.go:122] "RemoveContainer" containerID="f5dfc12cdf16ca735e2a0b2e1fc7623bdcf6ea3b577c7fd66a28132d83720e44" Nov 02 08:48:25.855732 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.855710 6160 scope.go:122] "RemoveContainer" containerID="fbab1e84bb3104efdea65a303a0541159ce5d24436f5d032235734fede1d7810" Nov 02 08:48:25.864483 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.864451 6160 scope.go:122] "RemoveContainer" containerID="9786f4c8cf0c5eb45703ffa571e30e054aeb6fe1b7ba3b974f384e60817c8bd9" Nov 02 08:48:25.989641 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:25.989610 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds"] Nov 02 08:48:26.035675 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:26.035637 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-mm2ds"] Nov 02 08:48:26.094624 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:26.094595 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p"] Nov 02 08:48:26.100103 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:26.100030 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rvn8p"] Nov 02 08:48:26.202238 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:26.202197 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd"] Nov 02 08:48:26.206769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:26.206737 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l54fd"] Nov 02 08:48:26.308362 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:26.308319 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb"] Nov 02 08:48:26.312197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:26.312152 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-x8nhb"] Nov 02 08:48:27.583539 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:27.583501 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" path="/var/lib/kubelet/pods/20d6fe81-2539-40b8-b08c-8ed5033da34c/volumes" Nov 02 08:48:27.583938 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:27.583922 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" path="/var/lib/kubelet/pods/73012162-12a0-467c-b849-1d9dd494c0b7/volumes" Nov 02 08:48:27.584341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:27.584328 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" path="/var/lib/kubelet/pods/c724ad29-9eb6-4e0c-9ff3-0f949e462de0/volumes" Nov 02 08:48:27.585030 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:27.585015 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" path="/var/lib/kubelet/pods/ec99e3d7-5c3d-4596-9b34-bdecbd94643f/volumes" Nov 02 08:48:27.843099 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:27.843009 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:28.408784 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:28.408748 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:28.587575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:28.587541 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:28.752545 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:28.752510 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:31.297911 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:31.297842 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:31.644322 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:31.644226 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:31.807442 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:31.807411 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:32.029491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.028217 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj"] Nov 02 08:48:32.029491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029169 6160 cpu_manager.go:414] "containerMap: removing container" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" Nov 02 08:48:32.029491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029211 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" Nov 02 08:48:32.029491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029224 6160 cpu_manager.go:414] "containerMap: removing container" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" Nov 02 08:48:32.029491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029231 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" Nov 02 08:48:32.029491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029270 6160 cpu_manager.go:414] "containerMap: removing container" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" Nov 02 08:48:32.029491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029278 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" Nov 02 08:48:32.029491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029352 6160 cpu_manager.go:414] "containerMap: removing container" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" Nov 02 08:48:32.029491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029360 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" Nov 02 08:48:32.030283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029759 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec99e3d7-5c3d-4596-9b34-bdecbd94643f" containerName="istio-proxy" Nov 02 08:48:32.030283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029777 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d6fe81-2539-40b8-b08c-8ed5033da34c" containerName="istio-proxy" Nov 02 08:48:32.030283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029789 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="73012162-12a0-467c-b849-1d9dd494c0b7" containerName="istio-proxy" Nov 02 08:48:32.030283 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.029809 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="c724ad29-9eb6-4e0c-9ff3-0f949e462de0" containerName="istio-proxy" Nov 02 08:48:32.036534 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.036508 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:32.084536 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.084502 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj"] Nov 02 08:48:32.127713 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.127670 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bbb1eb\"" Nov 02 08:48:32.127713 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.127675 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bbb1eb-dockercfg-k9qpt\"" Nov 02 08:48:32.134188 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.134138 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae84e55-efb9-452f-9f23-a13dc502760f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj\" (UID: \"eae84e55-efb9-452f-9f23-a13dc502760f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:32.134330 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.134316 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc5px\" (UniqueName: \"kubernetes.io/projected/eae84e55-efb9-452f-9f23-a13dc502760f-kube-api-access-gc5px\") pod \"payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj\" (UID: \"eae84e55-efb9-452f-9f23-a13dc502760f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:32.176100 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.176064 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb"] Nov 02 08:48:32.178760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.178741 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:32.225728 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.225681 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb"] Nov 02 08:48:32.235808 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.235775 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae84e55-efb9-452f-9f23-a13dc502760f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj\" (UID: \"eae84e55-efb9-452f-9f23-a13dc502760f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:32.235983 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.235875 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb\" (UID: \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:32.235983 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.235914 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrgpk\" (UniqueName: \"kubernetes.io/projected/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-kube-api-access-zrgpk\") pod \"payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb\" (UID: \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:32.236115 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.236061 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc5px\" (UniqueName: \"kubernetes.io/projected/eae84e55-efb9-452f-9f23-a13dc502760f-kube-api-access-gc5px\") pod \"payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj\" (UID: \"eae84e55-efb9-452f-9f23-a13dc502760f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:32.236598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.236574 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae84e55-efb9-452f-9f23-a13dc502760f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj\" (UID: \"eae84e55-efb9-452f-9f23-a13dc502760f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:32.290149 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.290078 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc5px\" (UniqueName: \"kubernetes.io/projected/eae84e55-efb9-452f-9f23-a13dc502760f-kube-api-access-gc5px\") pod \"payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj\" (UID: \"eae84e55-efb9-452f-9f23-a13dc502760f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:32.337644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.337531 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb\" (UID: \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:32.338068 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.337667 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgpk\" (UniqueName: \"kubernetes.io/projected/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-kube-api-access-zrgpk\") pod \"payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb\" (UID: \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:32.338214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.338193 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb\" (UID: \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:32.339491 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.339459 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:32.348341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.348307 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:32.391858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.391828 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrgpk\" (UniqueName: \"kubernetes.io/projected/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-kube-api-access-zrgpk\") pod \"payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb\" (UID: \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:32.492961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.492932 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:32.728557 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.728532 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj"] Nov 02 08:48:32.730389 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:48:32.730363 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae84e55_efb9_452f_9f23_a13dc502760f.slice/crio-e667b68a6ac12fc4b33ed37deb58820db7c1fad60413cfa24895de327af399a2 WatchSource:0}: Error finding container e667b68a6ac12fc4b33ed37deb58820db7c1fad60413cfa24895de327af399a2: Status 404 returned error can't find the container with id e667b68a6ac12fc4b33ed37deb58820db7c1fad60413cfa24895de327af399a2 Nov 02 08:48:32.844635 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.844539 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:32.883368 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.883333 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb"] Nov 02 08:48:32.886102 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:48:32.886077 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3ec2f3_6ff9_4070_9ecd_15ad7556596a.slice/crio-4b173d9ec5db9e009817b2d1d9ae3fea63a7d9df61363cdc796e0b94d7220ae6 WatchSource:0}: Error finding container 4b173d9ec5db9e009817b2d1d9ae3fea63a7d9df61363cdc796e0b94d7220ae6: Status 404 returned error can't find the container with id 4b173d9ec5db9e009817b2d1d9ae3fea63a7d9df61363cdc796e0b94d7220ae6 Nov 02 08:48:32.888141 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.888118 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" event={"ID":"eae84e55-efb9-452f-9f23-a13dc502760f","Type":"ContainerStarted","Data":"cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233"} Nov 02 08:48:32.888242 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.888160 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" event={"ID":"eae84e55-efb9-452f-9f23-a13dc502760f","Type":"ContainerStarted","Data":"e667b68a6ac12fc4b33ed37deb58820db7c1fad60413cfa24895de327af399a2"} Nov 02 08:48:32.888356 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.888288 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:32.994369 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.994308 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" podStartSLOduration=1.994293387 podStartE2EDuration="1.994293387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:48:32.992740729 +0000 UTC m=+2511.546244926" watchObservedRunningTime="2026-11-02 08:48:32.994293387 +0000 UTC m=+2511.547797630" Nov 02 08:48:32.995376 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:32.995332 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:33.895497 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:33.895457 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" event={"ID":"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a","Type":"ContainerStarted","Data":"f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51"} Nov 02 08:48:33.895908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:33.895668 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:33.895908 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:33.895705 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" event={"ID":"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a","Type":"ContainerStarted","Data":"4b173d9ec5db9e009817b2d1d9ae3fea63a7d9df61363cdc796e0b94d7220ae6"} Nov 02 08:48:34.004016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:34.003949 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" podStartSLOduration=2.003932582 podStartE2EDuration="2.003932582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:48:34.001693872 +0000 UTC m=+2512.555198074" watchObservedRunningTime="2026-11-02 08:48:34.003932582 +0000 UTC m=+2512.557436781" Nov 02 08:48:34.024554 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:34.024519 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:34.301810 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:34.301769 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:34.464718 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:34.464678 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:34.994408 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:34.994374 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:39.353396 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:39.353355 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:40.521755 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:48:40.521721 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:48:40.791671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:40.791583 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:41.012651 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:41.012619 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:41.172084 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:41.172046 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:41.648748 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:41.648663 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:43.899781 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:43.899747 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:48:44.372280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:44.372244 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:44.904929 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:44.904898 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:48:46.718601 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:46.718565 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:48.957986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:48.957954 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c"] Nov 02 08:48:48.962786 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:48.962767 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.011349 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.011317 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bbb1eb-serving-cert\"" Nov 02 08:48:49.060105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.060072 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c"] Nov 02 08:48:49.102972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.102941 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmn8h\" (UniqueName: \"kubernetes.io/projected/b5e203df-5b7b-481c-b5e3-50626c1c7ede-kube-api-access-tmn8h\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.103166 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.102978 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-metrics-certs\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.103166 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.103122 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.103348 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.103160 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e203df-5b7b-481c-b5e3-50626c1c7ede-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.183161 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.183125 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:49.204427 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.204386 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-metrics-certs\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.204623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.204510 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.204623 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.204551 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e203df-5b7b-481c-b5e3-50626c1c7ede-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.204762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.204655 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmn8h\" (UniqueName: \"kubernetes.io/projected/b5e203df-5b7b-481c-b5e3-50626c1c7ede-kube-api-access-tmn8h\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.204762 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:48:49.204704 6160 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-serving-cert: secret "maas-api-e2e-praxis-b-bbb1eb-serving-cert" not found Nov 02 08:48:49.204880 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:48:49.204794 6160 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls podName:b5e203df-5b7b-481c-b5e3-50626c1c7ede nodeName:}" failed. No retries permitted until 2026-11-02 08:48:49.704748991 +0000 UTC m=+2528.258253183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls") pod "maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" (UID: "b5e203df-5b7b-481c-b5e3-50626c1c7ede") : secret "maas-api-e2e-praxis-b-bbb1eb-serving-cert" not found Nov 02 08:48:49.205704 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.205677 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e203df-5b7b-481c-b5e3-50626c1c7ede-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.208145 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.208085 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-metrics-certs\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.259540 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.259505 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmn8h\" (UniqueName: \"kubernetes.io/projected/b5e203df-5b7b-481c-b5e3-50626c1c7ede-kube-api-access-tmn8h\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.448299 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.448266 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:49.710237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.710194 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.712709 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.712686 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:49.875538 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:49.875498 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:50.243094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:50.242985 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c"] Nov 02 08:48:50.245881 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:48:50.245847 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e203df_5b7b_481c_b5e3_50626c1c7ede.slice/crio-3d5a1bba3e5472404a6f224684c2538856012013ae0784e4e5e9142f319e9679 WatchSource:0}: Error finding container 3d5a1bba3e5472404a6f224684c2538856012013ae0784e4e5e9142f319e9679: Status 404 returned error can't find the container with id 3d5a1bba3e5472404a6f224684c2538856012013ae0784e4e5e9142f319e9679 Nov 02 08:48:50.996113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:50.996079 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" event={"ID":"b5e203df-5b7b-481c-b5e3-50626c1c7ede","Type":"ContainerStarted","Data":"3d5a1bba3e5472404a6f224684c2538856012013ae0784e4e5e9142f319e9679"} Nov 02 08:48:52.003967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:52.003933 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" event={"ID":"b5e203df-5b7b-481c-b5e3-50626c1c7ede","Type":"ContainerStarted","Data":"acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b"} Nov 02 08:48:52.004429 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:52.004065 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:52.114295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:52.112091 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" podStartSLOduration=3.238270906 podStartE2EDuration="4.112072363s" totalImagesPullingTime="873.801457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:48:52.110339339 +0000 UTC m=+2530.663843536" watchObservedRunningTime="2026-11-02 08:48:52.112072363 +0000 UTC m=+2530.665576564" Nov 02 08:48:54.153785 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:54.153753 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:56.155495 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:56.155454 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk"] Nov 02 08:48:56.264033 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:56.263995 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk"] Nov 02 08:48:56.565357 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:56.565305 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:56.721139 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:56.721103 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:58.015562 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:58.015518 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:48:58.952842 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:48:58.952807 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:01.155923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:01.155869 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" podUID="56e5538f-00d1-4b59-a85a-ec70ce18bf44" containerName="payload-processing" containerID="cri-o://c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905" gracePeriod=25 Nov 02 08:49:01.264576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:01.264531 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" podUID="1b02178e-9684-47f3-b053-cc9b96353d6a" containerName="payload-pre-processing" containerID="cri-o://3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e" gracePeriod=25 Nov 02 08:49:03.053457 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:03.053415 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:03.728227 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:03.728160 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:03.890597 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:03.890554 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:08.837200 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:08.837139 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:09.879160 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:09.879123 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" podUID="56e5538f-00d1-4b59-a85a-ec70ce18bf44" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:49:09.879592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:09.879126 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" podUID="1b02178e-9684-47f3-b053-cc9b96353d6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:49:10.520893 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:49:10.520855 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:49:10.923065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:10.923027 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:13.662958 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:13.662924 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:15.060960 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:15.060919 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:18.717494 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:18.717453 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:18.874116 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:18.874085 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:19.878625 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:19.878584 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" podUID="56e5538f-00d1-4b59-a85a-ec70ce18bf44" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:49:19.879016 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:19.878597 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" podUID="1b02178e-9684-47f3-b053-cc9b96353d6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:49:21.422754 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.422683 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:49:21.512274 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.512248 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:49:21.536239 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.536208 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e5538f-00d1-4b59-a85a-ec70ce18bf44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e5538f-00d1-4b59-a85a-ec70ce18bf44-plugins-config-volume\") pod \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\" (UID: \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\") " Nov 02 08:49:21.536412 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.536268 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e5538f-00d1-4b59-a85a-ec70ce18bf44-kube-api-access-cvb89\" (UniqueName: \"kubernetes.io/projected/56e5538f-00d1-4b59-a85a-ec70ce18bf44-kube-api-access-cvb89\") pod \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\" (UID: \"56e5538f-00d1-4b59-a85a-ec70ce18bf44\") " Nov 02 08:49:21.536560 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.536537 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e5538f-00d1-4b59-a85a-ec70ce18bf44-plugins-config-volume" pod "56e5538f-00d1-4b59-a85a-ec70ce18bf44" (UID: "56e5538f-00d1-4b59-a85a-ec70ce18bf44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:21.538609 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.538584 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e5538f-00d1-4b59-a85a-ec70ce18bf44-kube-api-access-cvb89" pod "56e5538f-00d1-4b59-a85a-ec70ce18bf44" (UID: "56e5538f-00d1-4b59-a85a-ec70ce18bf44"). InnerVolumeSpecName "kube-api-access-cvb89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:21.637619 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.637580 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b02178e-9684-47f3-b053-cc9b96353d6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02178e-9684-47f3-b053-cc9b96353d6a-plugins-config-volume\") pod \"1b02178e-9684-47f3-b053-cc9b96353d6a\" (UID: \"1b02178e-9684-47f3-b053-cc9b96353d6a\") " Nov 02 08:49:21.637619 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.637667 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b02178e-9684-47f3-b053-cc9b96353d6a-kube-api-access-z9rz2\" (UniqueName: \"kubernetes.io/projected/1b02178e-9684-47f3-b053-cc9b96353d6a-kube-api-access-z9rz2\") pod \"1b02178e-9684-47f3-b053-cc9b96353d6a\" (UID: \"1b02178e-9684-47f3-b053-cc9b96353d6a\") " Nov 02 08:49:21.638044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.638020 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvb89\" (UniqueName: \"kubernetes.io/projected/56e5538f-00d1-4b59-a85a-ec70ce18bf44-kube-api-access-cvb89\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:21.638128 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.638045 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e5538f-00d1-4b59-a85a-ec70ce18bf44-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:21.638128 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.638095 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b02178e-9684-47f3-b053-cc9b96353d6a-plugins-config-volume" pod "1b02178e-9684-47f3-b053-cc9b96353d6a" (UID: "1b02178e-9684-47f3-b053-cc9b96353d6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:21.640546 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.640522 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b02178e-9684-47f3-b053-cc9b96353d6a-kube-api-access-z9rz2" pod "1b02178e-9684-47f3-b053-cc9b96353d6a" (UID: "1b02178e-9684-47f3-b053-cc9b96353d6a"). InnerVolumeSpecName "kube-api-access-z9rz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:21.739277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.739167 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b02178e-9684-47f3-b053-cc9b96353d6a-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:21.739277 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:21.739225 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9rz2\" (UniqueName: \"kubernetes.io/projected/1b02178e-9684-47f3-b053-cc9b96353d6a-kube-api-access-z9rz2\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:22.171665 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.171627 6160 generic.go:356] "Generic (PLEG): container finished" podID="1b02178e-9684-47f3-b053-cc9b96353d6a" containerID="3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e" exitCode=0 Nov 02 08:49:22.171875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.171668 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" Nov 02 08:49:22.171875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.171715 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" event={"ID":"1b02178e-9684-47f3-b053-cc9b96353d6a","Type":"ContainerDied","Data":"3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e"} Nov 02 08:49:22.171875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.171770 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk" event={"ID":"1b02178e-9684-47f3-b053-cc9b96353d6a","Type":"ContainerDied","Data":"1c61ccf138e02500493a4dd51902e6d9514c22ce55b183e9cae9a40845b1b67d"} Nov 02 08:49:22.171875 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.171800 6160 scope.go:122] "RemoveContainer" containerID="3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e" Nov 02 08:49:22.173318 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.173294 6160 generic.go:356] "Generic (PLEG): container finished" podID="56e5538f-00d1-4b59-a85a-ec70ce18bf44" containerID="c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905" exitCode=0 Nov 02 08:49:22.173479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.173322 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" event={"ID":"56e5538f-00d1-4b59-a85a-ec70ce18bf44","Type":"ContainerDied","Data":"c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905"} Nov 02 08:49:22.173479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.173362 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" Nov 02 08:49:22.173479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.173384 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk" event={"ID":"56e5538f-00d1-4b59-a85a-ec70ce18bf44","Type":"ContainerDied","Data":"658e14a786f6706c1d386c343d2c5ae5f9495482c98574df33768d82b962c7ef"} Nov 02 08:49:22.185206 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.185086 6160 scope.go:122] "RemoveContainer" containerID="3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e" Nov 02 08:49:22.185615 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:49:22.185590 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e\": container with ID starting with 3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e not found: ID does not exist" containerID="3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e" Nov 02 08:49:22.185696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.185625 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e"} err="failed to get container status \"3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e\": rpc error: code = NotFound desc = could not find container \"3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e\": container with ID starting with 3d616f6d314e0f288b0be95417331ccefc419b05b23883740914e944a477617e not found: ID does not exist" Nov 02 08:49:22.185696 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.185648 6160 scope.go:122] "RemoveContainer" containerID="c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905" Nov 02 08:49:22.194458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.194437 6160 scope.go:122] "RemoveContainer" containerID="c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905" Nov 02 08:49:22.194717 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:49:22.194699 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905\": container with ID starting with c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905 not found: ID does not exist" containerID="c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905" Nov 02 08:49:22.194769 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.194724 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905"} err="failed to get container status \"c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905\": rpc error: code = NotFound desc = could not find container \"c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905\": container with ID starting with c11e7b6e3b0b3868c7d438211541e398ca4b945d4764ea9817fb20ef45f38905 not found: ID does not exist" Nov 02 08:49:22.324937 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.324901 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk"] Nov 02 08:49:22.328560 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.328537 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-096685-68c968b4c-nczhk"] Nov 02 08:49:22.429968 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.429881 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk"] Nov 02 08:49:22.432663 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:22.432637 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-096685-7b87f87d87-pxxvk"] Nov 02 08:49:23.580063 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:23.580022 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:23.584073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:23.584044 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b02178e-9684-47f3-b053-cc9b96353d6a" path="/var/lib/kubelet/pods/1b02178e-9684-47f3-b053-cc9b96353d6a/volumes" Nov 02 08:49:23.584904 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:23.584873 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e5538f-00d1-4b59-a85a-ec70ce18bf44" path="/var/lib/kubelet/pods/56e5538f-00d1-4b59-a85a-ec70ce18bf44/volumes" Nov 02 08:49:23.863936 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:23.863842 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:24.024954 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:24.024911 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:24.178474 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:24.178434 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:28.433916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:28.433876 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:29.606662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:29.606625 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:29.777493 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:29.777460 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:31.636105 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:31.636064 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:32.644652 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.644612 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt"] Nov 02 08:49:32.645165 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.644900 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" podUID="2db39649-c218-4d8b-ac5d-2c76d5ef092e" containerName="maas-api" containerID="cri-o://4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005" gracePeriod=30 Nov 02 08:49:32.775236 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.775190 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:32.903441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.903376 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:49:32.947838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.947809 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls\") pod \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " Nov 02 08:49:32.948006 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.947877 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db39649-c218-4d8b-ac5d-2c76d5ef092e-kube-api-access-zdf8x\" (UniqueName: \"kubernetes.io/projected/2db39649-c218-4d8b-ac5d-2c76d5ef092e-kube-api-access-zdf8x\") pod \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " Nov 02 08:49:32.948006 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.947912 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db39649-c218-4d8b-ac5d-2c76d5ef092e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db39649-c218-4d8b-ac5d-2c76d5ef092e-openshift-service-ca\") pod \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " Nov 02 08:49:32.948006 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.947951 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-metrics-certs\") pod \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\" (UID: \"2db39649-c218-4d8b-ac5d-2c76d5ef092e\") " Nov 02 08:49:32.948328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.948292 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db39649-c218-4d8b-ac5d-2c76d5ef092e-openshift-service-ca" pod "2db39649-c218-4d8b-ac5d-2c76d5ef092e" (UID: "2db39649-c218-4d8b-ac5d-2c76d5ef092e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:32.950289 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.950265 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db39649-c218-4d8b-ac5d-2c76d5ef092e-kube-api-access-zdf8x" pod "2db39649-c218-4d8b-ac5d-2c76d5ef092e" (UID: "2db39649-c218-4d8b-ac5d-2c76d5ef092e"). InnerVolumeSpecName "kube-api-access-zdf8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:32.950394 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.950326 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls" pod "2db39649-c218-4d8b-ac5d-2c76d5ef092e" (UID: "2db39649-c218-4d8b-ac5d-2c76d5ef092e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:32.950439 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:32.950388 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-metrics-certs" pod "2db39649-c218-4d8b-ac5d-2c76d5ef092e" (UID: "2db39649-c218-4d8b-ac5d-2c76d5ef092e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:33.049667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.049628 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdf8x\" (UniqueName: \"kubernetes.io/projected/2db39649-c218-4d8b-ac5d-2c76d5ef092e-kube-api-access-zdf8x\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:33.049667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.049659 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db39649-c218-4d8b-ac5d-2c76d5ef092e-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:33.049667 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.049671 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:33.049903 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.049684 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db39649-c218-4d8b-ac5d-2c76d5ef092e-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:33.241873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.241830 6160 generic.go:356] "Generic (PLEG): container finished" podID="2db39649-c218-4d8b-ac5d-2c76d5ef092e" containerID="4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005" exitCode=0 Nov 02 08:49:33.241873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.241874 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" Nov 02 08:49:33.242092 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.241884 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" event={"ID":"2db39649-c218-4d8b-ac5d-2c76d5ef092e","Type":"ContainerDied","Data":"4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005"} Nov 02 08:49:33.242092 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.241937 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt" event={"ID":"2db39649-c218-4d8b-ac5d-2c76d5ef092e","Type":"ContainerDied","Data":"63786873be90eeedd46e7d0eb3de11c8c53289adca6a732e79e9431dfd11dd48"} Nov 02 08:49:33.242092 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.241966 6160 scope.go:122] "RemoveContainer" containerID="4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005" Nov 02 08:49:33.252492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.252470 6160 scope.go:122] "RemoveContainer" containerID="4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005" Nov 02 08:49:33.252836 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:49:33.252815 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005\": container with ID starting with 4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005 not found: ID does not exist" containerID="4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005" Nov 02 08:49:33.252892 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.252844 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005"} err="failed to get container status \"4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005\": rpc error: code = NotFound desc = could not find container \"4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005\": container with ID starting with 4cac10540ace09f7695d382b8ffb44e0b07848d02d58bb4bf136b3cd617c3005 not found: ID does not exist" Nov 02 08:49:33.394966 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.394923 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt"] Nov 02 08:49:33.398224 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.398193 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-096685-68dc9bdb7-rrznt"] Nov 02 08:49:33.584833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:33.584759 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db39649-c218-4d8b-ac5d-2c76d5ef092e" path="/var/lib/kubelet/pods/2db39649-c218-4d8b-ac5d-2c76d5ef092e/volumes" Nov 02 08:49:34.438976 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:34.438941 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:34.763185 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:34.763069 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:34.928190 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:34.928148 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:35.330695 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:35.330654 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:39.922109 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:39.922073 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:40.521882 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:49:40.521841 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:49:41.042576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:41.042543 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:41.246405 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:41.246368 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:41.369147 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:41.369061 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn"] Nov 02 08:49:41.369315 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:41.369282 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" containerID="cri-o://7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f" gracePeriod=30 Nov 02 08:49:41.422654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:41.422624 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:41.925466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:41.925424 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g"] Nov 02 08:49:42.047547 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:42.047511 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp"] Nov 02 08:49:42.322972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:42.322879 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:42.522415 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:42.522374 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:44.850674 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:44.850639 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:46.617644 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.617615 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:49:46.693901 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.693861 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-credential-socket\") pod \"be98284b-e24f-4a50-bef2-c37d8f7dd924\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " Nov 02 08:49:46.694097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.693917 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-certs\") pod \"be98284b-e24f-4a50-bef2-c37d8f7dd924\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " Nov 02 08:49:46.694097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.693954 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-data\") pod \"be98284b-e24f-4a50-bef2-c37d8f7dd924\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " Nov 02 08:49:46.694097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.693971 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-socket\") pod \"be98284b-e24f-4a50-bef2-c37d8f7dd924\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " Nov 02 08:49:46.694097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694019 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-token\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-token\") pod \"be98284b-e24f-4a50-bef2-c37d8f7dd924\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " Nov 02 08:49:46.694097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694049 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-podinfo\") pod \"be98284b-e24f-4a50-bef2-c37d8f7dd924\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " Nov 02 08:49:46.694097 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694086 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-kube-api-access-g9l46\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-kube-api-access-g9l46\") pod \"be98284b-e24f-4a50-bef2-c37d8f7dd924\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " Nov 02 08:49:46.694458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694158 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-envoy\") pod \"be98284b-e24f-4a50-bef2-c37d8f7dd924\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " Nov 02 08:49:46.694458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694204 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-credential-socket" pod "be98284b-e24f-4a50-bef2-c37d8f7dd924" (UID: "be98284b-e24f-4a50-bef2-c37d8f7dd924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:46.694458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694275 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be98284b-e24f-4a50-bef2-c37d8f7dd924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be98284b-e24f-4a50-bef2-c37d8f7dd924-istiod-ca-cert\") pod \"be98284b-e24f-4a50-bef2-c37d8f7dd924\" (UID: \"be98284b-e24f-4a50-bef2-c37d8f7dd924\") " Nov 02 08:49:46.694458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694400 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-data" pod "be98284b-e24f-4a50-bef2-c37d8f7dd924" (UID: "be98284b-e24f-4a50-bef2-c37d8f7dd924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:46.694458 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694421 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-certs" pod "be98284b-e24f-4a50-bef2-c37d8f7dd924" (UID: "be98284b-e24f-4a50-bef2-c37d8f7dd924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:46.694722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694464 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-socket" pod "be98284b-e24f-4a50-bef2-c37d8f7dd924" (UID: "be98284b-e24f-4a50-bef2-c37d8f7dd924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:46.694722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694661 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:46.694722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694683 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:46.694722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694702 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:46.694722 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694718 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:46.695138 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.694683 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be98284b-e24f-4a50-bef2-c37d8f7dd924-istiod-ca-cert" pod "be98284b-e24f-4a50-bef2-c37d8f7dd924" (UID: "be98284b-e24f-4a50-bef2-c37d8f7dd924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:46.696654 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.696626 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-token" pod "be98284b-e24f-4a50-bef2-c37d8f7dd924" (UID: "be98284b-e24f-4a50-bef2-c37d8f7dd924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:46.696776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.696663 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-kube-api-access-g9l46" pod "be98284b-e24f-4a50-bef2-c37d8f7dd924" (UID: "be98284b-e24f-4a50-bef2-c37d8f7dd924"). InnerVolumeSpecName "kube-api-access-g9l46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:46.696776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.696664 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-podinfo" pod "be98284b-e24f-4a50-bef2-c37d8f7dd924" (UID: "be98284b-e24f-4a50-bef2-c37d8f7dd924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:49:46.696776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.696677 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-envoy" pod "be98284b-e24f-4a50-bef2-c37d8f7dd924" (UID: "be98284b-e24f-4a50-bef2-c37d8f7dd924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:49:46.795993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.795962 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:46.795993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.795993 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:46.795993 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.796004 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9l46\" (UniqueName: \"kubernetes.io/projected/be98284b-e24f-4a50-bef2-c37d8f7dd924-kube-api-access-g9l46\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:46.796298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.796019 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be98284b-e24f-4a50-bef2-c37d8f7dd924-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:46.796298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.796031 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be98284b-e24f-4a50-bef2-c37d8f7dd924-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:46.926384 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:46.926301 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" podUID="18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e" containerName="payload-processing" containerID="cri-o://3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478" gracePeriod=25 Nov 02 08:49:47.047852 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.047799 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" podUID="661e4ae8-b1e8-4985-8720-1d9401634a88" containerName="payload-pre-processing" containerID="cri-o://8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844" gracePeriod=25 Nov 02 08:49:47.166798 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.166778 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:49:47.289923 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.289895 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:49:47.301917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.301860 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-plugins-config-volume\") pod \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\" (UID: \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\") " Nov 02 08:49:47.301917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.301912 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-kube-api-access-zzqm6\" (UniqueName: \"kubernetes.io/projected/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-kube-api-access-zzqm6\") pod \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\" (UID: \"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e\") " Nov 02 08:49:47.302311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.302289 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-plugins-config-volume" pod "18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e" (UID: "18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:47.302388 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.302371 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:47.304128 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.304106 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-kube-api-access-zzqm6" pod "18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e" (UID: "18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e"). InnerVolumeSpecName "kube-api-access-zzqm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:47.322043 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.322005 6160 generic.go:356] "Generic (PLEG): container finished" podID="661e4ae8-b1e8-4985-8720-1d9401634a88" containerID="8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844" exitCode=0 Nov 02 08:49:47.322188 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.322048 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" Nov 02 08:49:47.322188 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.322113 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" event={"ID":"661e4ae8-b1e8-4985-8720-1d9401634a88","Type":"ContainerDied","Data":"8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844"} Nov 02 08:49:47.322188 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.322163 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp" event={"ID":"661e4ae8-b1e8-4985-8720-1d9401634a88","Type":"ContainerDied","Data":"bfe7ef5ec95770ef09f41cd9cc1216fcdd1de69f84ec26fac35921f998a9c4d2"} Nov 02 08:49:47.322340 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.322207 6160 scope.go:122] "RemoveContainer" containerID="8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844" Nov 02 08:49:47.323619 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.323593 6160 generic.go:356] "Generic (PLEG): container finished" podID="18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e" containerID="3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478" exitCode=0 Nov 02 08:49:47.323744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.323623 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" Nov 02 08:49:47.323744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.323677 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" event={"ID":"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e","Type":"ContainerDied","Data":"3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478"} Nov 02 08:49:47.323744 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.323725 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g" event={"ID":"18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e","Type":"ContainerDied","Data":"9e54bf78451ae73e8d680d9494f18af049eff9eb1e1e851d30a50890a94f16eb"} Nov 02 08:49:47.325113 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.325091 6160 generic.go:356] "Generic (PLEG): container finished" podID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerID="7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f" exitCode=0 Nov 02 08:49:47.325242 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.325124 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" event={"ID":"be98284b-e24f-4a50-bef2-c37d8f7dd924","Type":"ContainerDied","Data":"7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f"} Nov 02 08:49:47.325242 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.325148 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" Nov 02 08:49:47.325242 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.325192 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn" event={"ID":"be98284b-e24f-4a50-bef2-c37d8f7dd924","Type":"ContainerDied","Data":"080509ad96a238a0ef8c8698e648b33a018fca1e0ff5295e73a662cf4ebeef53"} Nov 02 08:49:47.332296 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.332234 6160 scope.go:122] "RemoveContainer" containerID="8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844" Nov 02 08:49:47.332610 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:49:47.332583 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844\": container with ID starting with 8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844 not found: ID does not exist" containerID="8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844" Nov 02 08:49:47.332700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.332619 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844"} err="failed to get container status \"8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844\": rpc error: code = NotFound desc = could not find container \"8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844\": container with ID starting with 8e50953af22e02436a9d4d561c20e3f93d27fc17bb56d587b374ca8f23634844 not found: ID does not exist" Nov 02 08:49:47.332700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.332642 6160 scope.go:122] "RemoveContainer" containerID="3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478" Nov 02 08:49:47.344057 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.344031 6160 scope.go:122] "RemoveContainer" containerID="3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478" Nov 02 08:49:47.344374 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:49:47.344352 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478\": container with ID starting with 3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478 not found: ID does not exist" containerID="3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478" Nov 02 08:49:47.344464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.344380 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478"} err="failed to get container status \"3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478\": rpc error: code = NotFound desc = could not find container \"3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478\": container with ID starting with 3346f329f56834293b8a2943cb37136d00992be66e147723923a8c1c1e006478 not found: ID does not exist" Nov 02 08:49:47.344464 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.344397 6160 scope.go:122] "RemoveContainer" containerID="7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f" Nov 02 08:49:47.353675 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.353655 6160 scope.go:122] "RemoveContainer" containerID="7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f" Nov 02 08:49:47.353967 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:49:47.353947 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f\": container with ID starting with 7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f not found: ID does not exist" containerID="7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f" Nov 02 08:49:47.354025 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.353980 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f"} err="failed to get container status \"7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f\": rpc error: code = NotFound desc = could not find container \"7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f\": container with ID starting with 7600b8cd85846e7100bf44bdbf42c3e9f90d0331ae2a7290ec3512281fea3c7f not found: ID does not exist" Nov 02 08:49:47.403693 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.403633 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661e4ae8-b1e8-4985-8720-1d9401634a88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661e4ae8-b1e8-4985-8720-1d9401634a88-plugins-config-volume\") pod \"661e4ae8-b1e8-4985-8720-1d9401634a88\" (UID: \"661e4ae8-b1e8-4985-8720-1d9401634a88\") " Nov 02 08:49:47.403866 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.403769 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661e4ae8-b1e8-4985-8720-1d9401634a88-kube-api-access-2xvds\" (UniqueName: \"kubernetes.io/projected/661e4ae8-b1e8-4985-8720-1d9401634a88-kube-api-access-2xvds\") pod \"661e4ae8-b1e8-4985-8720-1d9401634a88\" (UID: \"661e4ae8-b1e8-4985-8720-1d9401634a88\") " Nov 02 08:49:47.404062 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.404039 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661e4ae8-b1e8-4985-8720-1d9401634a88-plugins-config-volume" pod "661e4ae8-b1e8-4985-8720-1d9401634a88" (UID: "661e4ae8-b1e8-4985-8720-1d9401634a88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:47.404271 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.404250 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzqm6\" (UniqueName: \"kubernetes.io/projected/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e-kube-api-access-zzqm6\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:47.404328 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.404279 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/661e4ae8-b1e8-4985-8720-1d9401634a88-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:47.406341 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.406312 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661e4ae8-b1e8-4985-8720-1d9401634a88-kube-api-access-2xvds" pod "661e4ae8-b1e8-4985-8720-1d9401634a88" (UID: "661e4ae8-b1e8-4985-8720-1d9401634a88"). InnerVolumeSpecName "kube-api-access-2xvds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:47.505703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.505663 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xvds\" (UniqueName: \"kubernetes.io/projected/661e4ae8-b1e8-4985-8720-1d9401634a88-kube-api-access-2xvds\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:47.550668 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.550626 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn"] Nov 02 08:49:47.596738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.596703 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-096685-openshift-default-6ff6cb7b8f-bnbjn"] Nov 02 08:49:47.698223 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.698124 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g"] Nov 02 08:49:47.700219 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.700196 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8f1e34-6c697ddb45-jn68g"] Nov 02 08:49:47.801329 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.801292 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp"] Nov 02 08:49:47.807592 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:47.807566 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8f1e34-599f9f8f95-7ztcp"] Nov 02 08:49:49.582738 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:49.582703 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e" path="/var/lib/kubelet/pods/18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e/volumes" Nov 02 08:49:49.583168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:49.583140 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661e4ae8-b1e8-4985-8720-1d9401634a88" path="/var/lib/kubelet/pods/661e4ae8-b1e8-4985-8720-1d9401634a88/volumes" Nov 02 08:49:49.583595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:49.583569 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" path="/var/lib/kubelet/pods/be98284b-e24f-4a50-bef2-c37d8f7dd924/volumes" Nov 02 08:49:49.654056 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:49.654026 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:54.436707 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:54.436671 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:58.452056 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.452018 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b"] Nov 02 08:49:58.452540 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.452263 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" podUID="a47149d9-ef03-4b19-ae24-7215cdd1d93a" containerName="maas-api" containerID="cri-o://d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3" gracePeriod=30 Nov 02 08:49:58.589914 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.589875 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:58.699609 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.699581 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:49:58.817689 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.817606 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls\") pod \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " Nov 02 08:49:58.817689 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.817666 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47149d9-ef03-4b19-ae24-7215cdd1d93a-kube-api-access-sljtd\" (UniqueName: \"kubernetes.io/projected/a47149d9-ef03-4b19-ae24-7215cdd1d93a-kube-api-access-sljtd\") pod \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " Nov 02 08:49:58.817934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.817724 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-metrics-certs\") pod \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " Nov 02 08:49:58.817934 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.817791 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47149d9-ef03-4b19-ae24-7215cdd1d93a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47149d9-ef03-4b19-ae24-7215cdd1d93a-openshift-service-ca\") pod \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\" (UID: \"a47149d9-ef03-4b19-ae24-7215cdd1d93a\") " Nov 02 08:49:58.818240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.818212 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47149d9-ef03-4b19-ae24-7215cdd1d93a-openshift-service-ca" pod "a47149d9-ef03-4b19-ae24-7215cdd1d93a" (UID: "a47149d9-ef03-4b19-ae24-7215cdd1d93a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:58.820024 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.819989 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47149d9-ef03-4b19-ae24-7215cdd1d93a-kube-api-access-sljtd" pod "a47149d9-ef03-4b19-ae24-7215cdd1d93a" (UID: "a47149d9-ef03-4b19-ae24-7215cdd1d93a"). InnerVolumeSpecName "kube-api-access-sljtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:58.820484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.820456 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-metrics-certs" pod "a47149d9-ef03-4b19-ae24-7215cdd1d93a" (UID: "a47149d9-ef03-4b19-ae24-7215cdd1d93a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:58.820484 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.820475 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls" pod "a47149d9-ef03-4b19-ae24-7215cdd1d93a" (UID: "a47149d9-ef03-4b19-ae24-7215cdd1d93a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:58.918969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.918933 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:58.918969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.918965 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47149d9-ef03-4b19-ae24-7215cdd1d93a-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:58.918969 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.918976 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47149d9-ef03-4b19-ae24-7215cdd1d93a-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:58.919252 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:58.918984 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljtd\" (UniqueName: \"kubernetes.io/projected/a47149d9-ef03-4b19-ae24-7215cdd1d93a-kube-api-access-sljtd\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:49:59.401485 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:59.401440 6160 generic.go:356] "Generic (PLEG): container finished" podID="a47149d9-ef03-4b19-ae24-7215cdd1d93a" containerID="d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3" exitCode=0 Nov 02 08:49:59.401703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:59.401500 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" Nov 02 08:49:59.401703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:59.401516 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" event={"ID":"a47149d9-ef03-4b19-ae24-7215cdd1d93a","Type":"ContainerDied","Data":"d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3"} Nov 02 08:49:59.401703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:59.401572 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b" event={"ID":"a47149d9-ef03-4b19-ae24-7215cdd1d93a","Type":"ContainerDied","Data":"9a1bbd35c9ffccade78f90fa6149a98ccc6a1f625b90b67ef86c5e8cb8093bde"} Nov 02 08:49:59.401703 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:59.401594 6160 scope.go:122] "RemoveContainer" containerID="d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3" Nov 02 08:49:59.412782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:59.412763 6160 scope.go:122] "RemoveContainer" containerID="d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3" Nov 02 08:49:59.413067 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:49:59.413046 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3\": container with ID starting with d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3 not found: ID does not exist" containerID="d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3" Nov 02 08:49:59.413189 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:59.413071 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3"} err="failed to get container status \"d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3\": rpc error: code = NotFound desc = could not find container \"d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3\": container with ID starting with d2305149860bf38c83072c2a0fd1a66f28f834dacbf86200de112043ce0ef8f3 not found: ID does not exist" Nov 02 08:49:59.555724 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:59.555683 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b"] Nov 02 08:49:59.602193 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:49:59.602149 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8f1e34-796d97f5f7-f7h8b"] Nov 02 08:50:00.104331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:00.104295 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:00.373411 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:00.373317 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:00.534423 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:00.534388 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:01.027403 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:01.027367 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:01.592965 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:01.592931 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47149d9-ef03-4b19-ae24-7215cdd1d93a" path="/var/lib/kubelet/pods/a47149d9-ef03-4b19-ae24-7215cdd1d93a/volumes" Nov 02 08:50:05.558336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:05.558291 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:06.857348 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:06.857311 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:07.064632 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:07.064596 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:07.153504 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:07.153420 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs"] Nov 02 08:50:07.153662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:07.153643 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" containerID="cri-o://0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d" gracePeriod=30 Nov 02 08:50:07.220661 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:07.220624 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:10.510244 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:10.510207 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:10.520917 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:50:10.520884 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:50:11.313872 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:11.313834 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:12.398100 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.398070 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:50:12.438455 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438375 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-envoy\") pod \"95021cec-10e1-4c10-b122-999e183abde6\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " Nov 02 08:50:12.438455 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438422 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-certs\") pod \"95021cec-10e1-4c10-b122-999e183abde6\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " Nov 02 08:50:12.438662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438478 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95021cec-10e1-4c10-b122-999e183abde6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95021cec-10e1-4c10-b122-999e183abde6-istio-podinfo\") pod \"95021cec-10e1-4c10-b122-999e183abde6\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " Nov 02 08:50:12.438662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438549 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-istio-token\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-istio-token\") pod \"95021cec-10e1-4c10-b122-999e183abde6\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " Nov 02 08:50:12.438662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438582 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-kube-api-access-rhmdg\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-kube-api-access-rhmdg\") pod \"95021cec-10e1-4c10-b122-999e183abde6\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " Nov 02 08:50:12.438662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438618 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-socket\") pod \"95021cec-10e1-4c10-b122-999e183abde6\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " Nov 02 08:50:12.438662 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438648 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95021cec-10e1-4c10-b122-999e183abde6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95021cec-10e1-4c10-b122-999e183abde6-istiod-ca-cert\") pod \"95021cec-10e1-4c10-b122-999e183abde6\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " Nov 02 08:50:12.438917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438684 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-credential-socket\") pod \"95021cec-10e1-4c10-b122-999e183abde6\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " Nov 02 08:50:12.438917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438702 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-certs" pod "95021cec-10e1-4c10-b122-999e183abde6" (UID: "95021cec-10e1-4c10-b122-999e183abde6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:12.438917 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.438719 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-data\") pod \"95021cec-10e1-4c10-b122-999e183abde6\" (UID: \"95021cec-10e1-4c10-b122-999e183abde6\") " Nov 02 08:50:12.439074 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.439013 6160 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.439157 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.439130 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-credential-socket" pod "95021cec-10e1-4c10-b122-999e183abde6" (UID: "95021cec-10e1-4c10-b122-999e183abde6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:12.439273 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.439250 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-data" pod "95021cec-10e1-4c10-b122-999e183abde6" (UID: "95021cec-10e1-4c10-b122-999e183abde6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:12.439336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.439268 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-socket" pod "95021cec-10e1-4c10-b122-999e183abde6" (UID: "95021cec-10e1-4c10-b122-999e183abde6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:12.439336 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.439254 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95021cec-10e1-4c10-b122-999e183abde6-istiod-ca-cert" pod "95021cec-10e1-4c10-b122-999e183abde6" (UID: "95021cec-10e1-4c10-b122-999e183abde6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:12.440760 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.440741 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-envoy" pod "95021cec-10e1-4c10-b122-999e183abde6" (UID: "95021cec-10e1-4c10-b122-999e183abde6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:12.440979 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.440961 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-kube-api-access-rhmdg" pod "95021cec-10e1-4c10-b122-999e183abde6" (UID: "95021cec-10e1-4c10-b122-999e183abde6"). InnerVolumeSpecName "kube-api-access-rhmdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:12.441281 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.441262 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95021cec-10e1-4c10-b122-999e183abde6-istio-podinfo" pod "95021cec-10e1-4c10-b122-999e183abde6" (UID: "95021cec-10e1-4c10-b122-999e183abde6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:50:12.441353 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.441282 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-istio-token" pod "95021cec-10e1-4c10-b122-999e183abde6" (UID: "95021cec-10e1-4c10-b122-999e183abde6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:12.473603 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.473575 6160 generic.go:356] "Generic (PLEG): container finished" podID="95021cec-10e1-4c10-b122-999e183abde6" containerID="0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d" exitCode=0 Nov 02 08:50:12.473731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.473609 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" event={"ID":"95021cec-10e1-4c10-b122-999e183abde6","Type":"ContainerDied","Data":"0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d"} Nov 02 08:50:12.473731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.473630 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" Nov 02 08:50:12.473731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.473669 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs" event={"ID":"95021cec-10e1-4c10-b122-999e183abde6","Type":"ContainerDied","Data":"4ee710901f1827d873c385beeb967150d29f01b1c4724deead19fa3fe529ea9e"} Nov 02 08:50:12.473731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.473692 6160 scope.go:122] "RemoveContainer" containerID="0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d" Nov 02 08:50:12.483360 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.483328 6160 scope.go:122] "RemoveContainer" containerID="a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff" Nov 02 08:50:12.491991 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.491974 6160 scope.go:122] "RemoveContainer" containerID="0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d" Nov 02 08:50:12.492285 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:50:12.492266 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d\": container with ID starting with 0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d not found: ID does not exist" containerID="0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d" Nov 02 08:50:12.492343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.492297 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d"} err="failed to get container status \"0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d\": rpc error: code = NotFound desc = could not find container \"0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d\": container with ID starting with 0e927df009d4c2389f00c53190ef22070fd9b491b9bf1ba8154f75a474164b8d not found: ID does not exist" Nov 02 08:50:12.492343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.492318 6160 scope.go:122] "RemoveContainer" containerID="a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff" Nov 02 08:50:12.492551 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:50:12.492535 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff\": container with ID starting with a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff not found: ID does not exist" containerID="a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff" Nov 02 08:50:12.492618 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.492554 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff"} err="failed to get container status \"a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff\": rpc error: code = NotFound desc = could not find container \"a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff\": container with ID starting with a7aba0d5df4be6d798f929d8df8ef36630a95ec0d6ee86144928454a9846b4ff not found: ID does not exist" Nov 02 08:50:12.540005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.539973 6160 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-istio-token\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.540005 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.539999 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmdg\" (UniqueName: \"kubernetes.io/projected/95021cec-10e1-4c10-b122-999e183abde6-kube-api-access-rhmdg\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.540214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.540016 6160 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-workload-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.540214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.540029 6160 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95021cec-10e1-4c10-b122-999e183abde6-istiod-ca-cert\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.540214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.540041 6160 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-credential-socket\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.540214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.540052 6160 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-data\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.540214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.540066 6160 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95021cec-10e1-4c10-b122-999e183abde6-istio-envoy\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.540214 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.540078 6160 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95021cec-10e1-4c10-b122-999e183abde6-istio-podinfo\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:12.658306 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.658275 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs"] Nov 02 08:50:12.663942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:12.663916 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-8f1e34-openshift-default-b8cf56fd9-lzgrs"] Nov 02 08:50:13.583000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:13.582962 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95021cec-10e1-4c10-b122-999e183abde6" path="/var/lib/kubelet/pods/95021cec-10e1-4c10-b122-999e183abde6/volumes" Nov 02 08:50:15.409204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:15.409135 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:15.612202 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:15.612141 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:19.436566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.436526 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb"] Nov 02 08:50:19.551742 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.551702 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj"] Nov 02 08:50:19.617701 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.617670 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c"] Nov 02 08:50:19.617887 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.617862 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" podUID="b5e203df-5b7b-481c-b5e3-50626c1c7ede" containerName="maas-api" containerID="cri-o://acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b" gracePeriod=30 Nov 02 08:50:19.747343 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.747306 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:19.870256 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.870228 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:50:19.911166 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.911135 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e203df-5b7b-481c-b5e3-50626c1c7ede-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e203df-5b7b-481c-b5e3-50626c1c7ede-openshift-service-ca\") pod \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " Nov 02 08:50:19.911338 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.911278 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-metrics-certs\") pod \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " Nov 02 08:50:19.911338 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.911323 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls\") pod \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " Nov 02 08:50:19.911482 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.911347 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e203df-5b7b-481c-b5e3-50626c1c7ede-kube-api-access-tmn8h\" (UniqueName: \"kubernetes.io/projected/b5e203df-5b7b-481c-b5e3-50626c1c7ede-kube-api-access-tmn8h\") pod \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\" (UID: \"b5e203df-5b7b-481c-b5e3-50626c1c7ede\") " Nov 02 08:50:19.911544 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.911479 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e203df-5b7b-481c-b5e3-50626c1c7ede-openshift-service-ca" pod "b5e203df-5b7b-481c-b5e3-50626c1c7ede" (UID: "b5e203df-5b7b-481c-b5e3-50626c1c7ede"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:19.913614 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.913587 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-metrics-certs" pod "b5e203df-5b7b-481c-b5e3-50626c1c7ede" (UID: "b5e203df-5b7b-481c-b5e3-50626c1c7ede"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:19.913726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.913621 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls" pod "b5e203df-5b7b-481c-b5e3-50626c1c7ede" (UID: "b5e203df-5b7b-481c-b5e3-50626c1c7ede"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:19.913726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.913641 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e203df-5b7b-481c-b5e3-50626c1c7ede-kube-api-access-tmn8h" pod "b5e203df-5b7b-481c-b5e3-50626c1c7ede" (UID: "b5e203df-5b7b-481c-b5e3-50626c1c7ede"). InnerVolumeSpecName "kube-api-access-tmn8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:19.952895 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:19.952857 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:20.012863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.012780 6160 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-metrics-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:20.012863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.012812 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e203df-5b7b-481c-b5e3-50626c1c7ede-maas-api-tls\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:20.012863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.012824 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmn8h\" (UniqueName: \"kubernetes.io/projected/b5e203df-5b7b-481c-b5e3-50626c1c7ede-kube-api-access-tmn8h\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:20.012863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.012834 6160 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e203df-5b7b-481c-b5e3-50626c1c7ede-openshift-service-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:20.099858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.099815 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:20.518361 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.518325 6160 generic.go:356] "Generic (PLEG): container finished" podID="b5e203df-5b7b-481c-b5e3-50626c1c7ede" containerID="acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b" exitCode=0 Nov 02 08:50:20.518361 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.518373 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" Nov 02 08:50:20.519020 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.518381 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" event={"ID":"b5e203df-5b7b-481c-b5e3-50626c1c7ede","Type":"ContainerDied","Data":"acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b"} Nov 02 08:50:20.519020 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.518433 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c" event={"ID":"b5e203df-5b7b-481c-b5e3-50626c1c7ede","Type":"ContainerDied","Data":"3d5a1bba3e5472404a6f224684c2538856012013ae0784e4e5e9142f319e9679"} Nov 02 08:50:20.519020 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.518466 6160 scope.go:122] "RemoveContainer" containerID="acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b" Nov 02 08:50:20.529080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.529061 6160 scope.go:122] "RemoveContainer" containerID="acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b" Nov 02 08:50:20.529381 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:50:20.529360 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b\": container with ID starting with acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b not found: ID does not exist" containerID="acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b" Nov 02 08:50:20.529443 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.529390 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b"} err="failed to get container status \"acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b\": rpc error: code = NotFound desc = could not find container \"acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b\": container with ID starting with acebfcdf962246f759432faab911432000ae0c49d082e624dfda99b9761f941b not found: ID does not exist" Nov 02 08:50:20.672280 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.672238 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c"] Nov 02 08:50:20.677274 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:20.677242 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bbb1eb-78748949f-6gj5c"] Nov 02 08:50:21.538503 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:21.538469 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:21.583492 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:21.583462 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e203df-5b7b-481c-b5e3-50626c1c7ede" path="/var/lib/kubelet/pods/b5e203df-5b7b-481c-b5e3-50626c1c7ede/volumes" Nov 02 08:50:23.939258 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:23.939218 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:24.270610 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.270514 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:24.437766 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.437716 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" podUID="ad3ec2f3-6ff9-4070-9ecd-15ad7556596a" containerName="payload-processing" containerID="cri-o://f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51" gracePeriod=25 Nov 02 08:50:24.552573 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.552477 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" podUID="eae84e55-efb9-452f-9f23-a13dc502760f" containerName="payload-pre-processing" containerID="cri-o://cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233" gracePeriod=25 Nov 02 08:50:24.711363 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.711315 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:50:24.756671 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.756641 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-kube-api-access-zrgpk\" (UniqueName: \"kubernetes.io/projected/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-kube-api-access-zrgpk\") pod \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\" (UID: \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\") " Nov 02 08:50:24.756837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.756718 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-plugins-config-volume\") pod \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\" (UID: \"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a\") " Nov 02 08:50:24.757102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.757077 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-plugins-config-volume" pod "ad3ec2f3-6ff9-4070-9ecd-15ad7556596a" (UID: "ad3ec2f3-6ff9-4070-9ecd-15ad7556596a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:24.759288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.759262 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-kube-api-access-zrgpk" pod "ad3ec2f3-6ff9-4070-9ecd-15ad7556596a" (UID: "ad3ec2f3-6ff9-4070-9ecd-15ad7556596a"). InnerVolumeSpecName "kube-api-access-zrgpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:24.817971 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.817941 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:50:24.857505 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.857474 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae84e55-efb9-452f-9f23-a13dc502760f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae84e55-efb9-452f-9f23-a13dc502760f-plugins-config-volume\") pod \"eae84e55-efb9-452f-9f23-a13dc502760f\" (UID: \"eae84e55-efb9-452f-9f23-a13dc502760f\") " Nov 02 08:50:24.857705 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.857581 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae84e55-efb9-452f-9f23-a13dc502760f-kube-api-access-gc5px\" (UniqueName: \"kubernetes.io/projected/eae84e55-efb9-452f-9f23-a13dc502760f-kube-api-access-gc5px\") pod \"eae84e55-efb9-452f-9f23-a13dc502760f\" (UID: \"eae84e55-efb9-452f-9f23-a13dc502760f\") " Nov 02 08:50:24.857882 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.857863 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrgpk\" (UniqueName: \"kubernetes.io/projected/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-kube-api-access-zrgpk\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:24.857942 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.857894 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:24.858001 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.857973 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae84e55-efb9-452f-9f23-a13dc502760f-plugins-config-volume" pod "eae84e55-efb9-452f-9f23-a13dc502760f" (UID: "eae84e55-efb9-452f-9f23-a13dc502760f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:24.859837 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.859815 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae84e55-efb9-452f-9f23-a13dc502760f-kube-api-access-gc5px" pod "eae84e55-efb9-452f-9f23-a13dc502760f" (UID: "eae84e55-efb9-452f-9f23-a13dc502760f"). InnerVolumeSpecName "kube-api-access-gc5px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:24.871048 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.871023 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:24.959197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.959134 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc5px\" (UniqueName: \"kubernetes.io/projected/eae84e55-efb9-452f-9f23-a13dc502760f-kube-api-access-gc5px\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:24.959197 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:24.959167 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae84e55-efb9-452f-9f23-a13dc502760f-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:25.547267 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.547233 6160 generic.go:356] "Generic (PLEG): container finished" podID="ad3ec2f3-6ff9-4070-9ecd-15ad7556596a" containerID="f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51" exitCode=0 Nov 02 08:50:25.547438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.547299 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" Nov 02 08:50:25.547438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.547321 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" event={"ID":"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a","Type":"ContainerDied","Data":"f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51"} Nov 02 08:50:25.547438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.547403 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb" event={"ID":"ad3ec2f3-6ff9-4070-9ecd-15ad7556596a","Type":"ContainerDied","Data":"4b173d9ec5db9e009817b2d1d9ae3fea63a7d9df61363cdc796e0b94d7220ae6"} Nov 02 08:50:25.547438 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.547433 6160 scope.go:122] "RemoveContainer" containerID="f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51" Nov 02 08:50:25.548788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.548767 6160 generic.go:356] "Generic (PLEG): container finished" podID="eae84e55-efb9-452f-9f23-a13dc502760f" containerID="cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233" exitCode=0 Nov 02 08:50:25.548897 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.548816 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" event={"ID":"eae84e55-efb9-452f-9f23-a13dc502760f","Type":"ContainerDied","Data":"cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233"} Nov 02 08:50:25.548897 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.548822 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" Nov 02 08:50:25.548897 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.548857 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj" event={"ID":"eae84e55-efb9-452f-9f23-a13dc502760f","Type":"ContainerDied","Data":"e667b68a6ac12fc4b33ed37deb58820db7c1fad60413cfa24895de327af399a2"} Nov 02 08:50:25.557067 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.557044 6160 scope.go:122] "RemoveContainer" containerID="f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51" Nov 02 08:50:25.557353 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:50:25.557332 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51\": container with ID starting with f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51 not found: ID does not exist" containerID="f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51" Nov 02 08:50:25.557460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.557363 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51"} err="failed to get container status \"f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51\": rpc error: code = NotFound desc = could not find container \"f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51\": container with ID starting with f763170beddda60d9168f1fdb9be2c0e29ab66b9d39c5d71a0f35ff497ec1f51 not found: ID does not exist" Nov 02 08:50:25.557460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.557390 6160 scope.go:122] "RemoveContainer" containerID="cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233" Nov 02 08:50:25.565550 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.565533 6160 scope.go:122] "RemoveContainer" containerID="cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233" Nov 02 08:50:25.565811 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:50:25.565794 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233\": container with ID starting with cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233 not found: ID does not exist" containerID="cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233" Nov 02 08:50:25.565868 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.565818 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233"} err="failed to get container status \"cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233\": rpc error: code = NotFound desc = could not find container \"cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233\": container with ID starting with cb08d217a8a801f3c7704510ae9cb25e267136a243502597cf8117aaebaa6233 not found: ID does not exist" Nov 02 08:50:25.701496 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.701388 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj"] Nov 02 08:50:25.705146 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.705117 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bbb1eb-85c98c7c7f-f8gpj"] Nov 02 08:50:25.807498 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.807450 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb"] Nov 02 08:50:25.809404 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:25.809380 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bbb1eb-9df7698b5-fvnsb"] Nov 02 08:50:26.576335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:26.576299 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:27.582817 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:27.582784 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3ec2f3-6ff9-4070-9ecd-15ad7556596a" path="/var/lib/kubelet/pods/ad3ec2f3-6ff9-4070-9ecd-15ad7556596a/volumes" Nov 02 08:50:27.583225 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:27.583168 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae84e55-efb9-452f-9f23-a13dc502760f" path="/var/lib/kubelet/pods/eae84e55-efb9-452f-9f23-a13dc502760f/volumes" Nov 02 08:50:30.575378 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:30.575342 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:30.753827 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:30.753791 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:30.926977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:30.926938 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:31.388288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.388202 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:31.591630 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.591597 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:31.902791 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.902758 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d"] Nov 02 08:50:31.903116 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903104 6160 cpu_manager.go:414] "containerMap: removing container" podUID="a47149d9-ef03-4b19-ae24-7215cdd1d93a" containerName="maas-api" Nov 02 08:50:31.903163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903118 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47149d9-ef03-4b19-ae24-7215cdd1d93a" containerName="maas-api" Nov 02 08:50:31.903163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903127 6160 cpu_manager.go:414] "containerMap: removing container" podUID="eae84e55-efb9-452f-9f23-a13dc502760f" containerName="payload-pre-processing" Nov 02 08:50:31.903163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903132 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae84e55-efb9-452f-9f23-a13dc502760f" containerName="payload-pre-processing" Nov 02 08:50:31.903163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903150 6160 cpu_manager.go:414] "containerMap: removing container" podUID="ad3ec2f3-6ff9-4070-9ecd-15ad7556596a" containerName="payload-processing" Nov 02 08:50:31.903163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903155 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3ec2f3-6ff9-4070-9ecd-15ad7556596a" containerName="payload-processing" Nov 02 08:50:31.903163 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903162 6160 cpu_manager.go:414] "containerMap: removing container" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903168 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903195 6160 cpu_manager.go:414] "containerMap: removing container" podUID="b5e203df-5b7b-481c-b5e3-50626c1c7ede" containerName="maas-api" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903200 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e203df-5b7b-481c-b5e3-50626c1c7ede" containerName="maas-api" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903206 6160 cpu_manager.go:414] "containerMap: removing container" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903210 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903218 6160 cpu_manager.go:414] "containerMap: removing container" podUID="18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e" containerName="payload-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903222 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e" containerName="payload-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903227 6160 cpu_manager.go:414] "containerMap: removing container" podUID="661e4ae8-b1e8-4985-8720-1d9401634a88" containerName="payload-pre-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903231 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661e4ae8-b1e8-4985-8720-1d9401634a88" containerName="payload-pre-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903238 6160 cpu_manager.go:414] "containerMap: removing container" podUID="2db39649-c218-4d8b-ac5d-2c76d5ef092e" containerName="maas-api" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903244 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db39649-c218-4d8b-ac5d-2c76d5ef092e" containerName="maas-api" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903251 6160 cpu_manager.go:414] "containerMap: removing container" podUID="1b02178e-9684-47f3-b053-cc9b96353d6a" containerName="payload-pre-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903255 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b02178e-9684-47f3-b053-cc9b96353d6a" containerName="payload-pre-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903262 6160 cpu_manager.go:414] "containerMap: removing container" podUID="56e5538f-00d1-4b59-a85a-ec70ce18bf44" containerName="payload-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903267 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e5538f-00d1-4b59-a85a-ec70ce18bf44" containerName="payload-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903276 6160 cpu_manager.go:414] "containerMap: removing container" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903280 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903328 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903335 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e203df-5b7b-481c-b5e3-50626c1c7ede" containerName="maas-api" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903341 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="be98284b-e24f-4a50-bef2-c37d8f7dd924" containerName="istio-proxy" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903347 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="18eabf3d-e6cb-41d9-9a0e-f3ce8b6e853e" containerName="payload-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903351 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="661e4ae8-b1e8-4985-8720-1d9401634a88" containerName="payload-pre-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903357 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db39649-c218-4d8b-ac5d-2c76d5ef092e" containerName="maas-api" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903362 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b02178e-9684-47f3-b053-cc9b96353d6a" containerName="payload-pre-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903367 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e5538f-00d1-4b59-a85a-ec70ce18bf44" containerName="payload-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903372 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="95021cec-10e1-4c10-b122-999e183abde6" containerName="istio-proxy" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903378 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47149d9-ef03-4b19-ae24-7215cdd1d93a" containerName="maas-api" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903383 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae84e55-efb9-452f-9f23-a13dc502760f" containerName="payload-pre-processing" Nov 02 08:50:31.903377 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.903390 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3ec2f3-6ff9-4070-9ecd-15ad7556596a" containerName="payload-processing" Nov 02 08:50:31.908037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.908020 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:31.960452 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.960419 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:50:31.961120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.961096 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d"] Nov 02 08:50:31.982168 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:31.982135 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:32.005027 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.004986 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:50:32.005027 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.005001 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:50:32.005239 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.005007 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:50:32.024023 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.023984 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxczh\" (UniqueName: \"kubernetes.io/projected/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-kube-api-access-jxczh\") pod \"maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.024023 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.024024 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.024310 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.024147 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.125035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.125001 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxczh\" (UniqueName: \"kubernetes.io/projected/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-kube-api-access-jxczh\") pod \"maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.125035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.125035 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.125333 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.125079 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.125762 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.125735 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.127585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.127566 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.178432 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.178392 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxczh\" (UniqueName: \"kubernetes.io/projected/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-kube-api-access-jxczh\") pod \"maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.221021 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.220986 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:32.589997 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:32.589969 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d"] Nov 02 08:50:32.592140 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:50:32.592109 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396ea774_88ee_46b2_9b3b_ad6bc9b686ee.slice/crio-3d79cd26e29558ba21c4e25c7d69e852fc23e0a0372a5d6bca1786c40415abd5 WatchSource:0}: Error finding container 3d79cd26e29558ba21c4e25c7d69e852fc23e0a0372a5d6bca1786c40415abd5: Status 404 returned error can't find the container with id 3d79cd26e29558ba21c4e25c7d69e852fc23e0a0372a5d6bca1786c40415abd5 Nov 02 08:50:33.604890 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:33.604852 6160 generic.go:356] "Generic (PLEG): container finished" podID="396ea774-88ee-46b2-9b3b-ad6bc9b686ee" containerID="5fac2800551e692981e4f3c067d403f2e456a3a9a25c082e2c06c93826fbc19e" exitCode=0 Nov 02 08:50:33.605268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:33.604939 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" event={"ID":"396ea774-88ee-46b2-9b3b-ad6bc9b686ee","Type":"ContainerDied","Data":"5fac2800551e692981e4f3c067d403f2e456a3a9a25c082e2c06c93826fbc19e"} Nov 02 08:50:33.605268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:33.604993 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" event={"ID":"396ea774-88ee-46b2-9b3b-ad6bc9b686ee","Type":"ContainerStarted","Data":"3d79cd26e29558ba21c4e25c7d69e852fc23e0a0372a5d6bca1786c40415abd5"} Nov 02 08:50:34.750237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.750140 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:34.853042 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.853007 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-openshift-ca\") pod \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " Nov 02 08:50:34.853226 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.853049 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-xks-ca\") pod \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " Nov 02 08:50:34.853272 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.853241 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-kube-api-access-jxczh\" (UniqueName: \"kubernetes.io/projected/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-kube-api-access-jxczh\") pod \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\" (UID: \"396ea774-88ee-46b2-9b3b-ad6bc9b686ee\") " Nov 02 08:50:34.853475 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.853446 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-openshift-ca" pod "396ea774-88ee-46b2-9b3b-ad6bc9b686ee" (UID: "396ea774-88ee-46b2-9b3b-ad6bc9b686ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:34.853575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.853548 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-openshift-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:34.855323 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.855301 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-xks-ca" pod "396ea774-88ee-46b2-9b3b-ad6bc9b686ee" (UID: "396ea774-88ee-46b2-9b3b-ad6bc9b686ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:34.855479 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.855458 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-kube-api-access-jxczh" pod "396ea774-88ee-46b2-9b3b-ad6bc9b686ee" (UID: "396ea774-88ee-46b2-9b3b-ad6bc9b686ee"). InnerVolumeSpecName "kube-api-access-jxczh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:34.954278 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.954243 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-maas-api-xks-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:34.954278 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:34.954275 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxczh\" (UniqueName: \"kubernetes.io/projected/396ea774-88ee-46b2-9b3b-ad6bc9b686ee-kube-api-access-jxczh\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:35.617370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:35.617336 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" event={"ID":"396ea774-88ee-46b2-9b3b-ad6bc9b686ee","Type":"ContainerDied","Data":"3d79cd26e29558ba21c4e25c7d69e852fc23e0a0372a5d6bca1786c40415abd5"} Nov 02 08:50:35.617370 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:35.617365 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d" Nov 02 08:50:35.617585 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:35.617382 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d79cd26e29558ba21c4e25c7d69e852fc23e0a0372a5d6bca1786c40415abd5" Nov 02 08:50:36.364999 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:36.364965 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv"] Nov 02 08:50:36.542970 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:36.542932 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:36.772883 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:36.772844 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:40.520479 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:50:40.520441 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:50:41.651415 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:41.651381 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:43.499305 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:43.499259 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw"] Nov 02 08:50:43.776591 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:43.776507 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:43.936363 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:43.936319 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:46.342246 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:46.342211 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:48.499776 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:48.499729 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" podUID="84488f2a-6ede-40e9-a394-2db0a6f25669" containerName="payload-pre-processing" containerID="cri-o://da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75" gracePeriod=25 Nov 02 08:50:49.400409 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:49.400364 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:49.579918 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:49.579866 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:50.533928 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.533891 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm"] Nov 02 08:50:50.534307 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.534293 6160 cpu_manager.go:414] "containerMap: removing container" podUID="396ea774-88ee-46b2-9b3b-ad6bc9b686ee" containerName="revoke-keys" Nov 02 08:50:50.534359 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.534307 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396ea774-88ee-46b2-9b3b-ad6bc9b686ee" containerName="revoke-keys" Nov 02 08:50:50.534395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.534369 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="396ea774-88ee-46b2-9b3b-ad6bc9b686ee" containerName="revoke-keys" Nov 02 08:50:50.539203 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.539160 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.591847 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.591816 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 08:50:50.606295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.606267 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b06bad-rsbbm\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.606441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.606394 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b06bad-rsbbm\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.606441 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.606435 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95vqw\" (UniqueName: \"kubernetes.io/projected/e485365b-0cb4-440b-aa4d-64a58558cd15-kube-api-access-95vqw\") pod \"maas-api-revoke-sub-d9b06bad-rsbbm\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.634206 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.634159 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:50:50.634342 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.634218 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4p9nt\"" Nov 02 08:50:50.634342 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.634233 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 08:50:50.636466 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.636444 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm"] Nov 02 08:50:50.707220 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.707159 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95vqw\" (UniqueName: \"kubernetes.io/projected/e485365b-0cb4-440b-aa4d-64a58558cd15-kube-api-access-95vqw\") pod \"maas-api-revoke-sub-d9b06bad-rsbbm\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.707395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.707224 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b06bad-rsbbm\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.707395 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.707321 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b06bad-rsbbm\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.708065 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.707992 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b06bad-rsbbm\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.709806 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.709774 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b06bad-rsbbm\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.760375 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.760347 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95vqw\" (UniqueName: \"kubernetes.io/projected/e485365b-0cb4-440b-aa4d-64a58558cd15-kube-api-access-95vqw\") pod \"maas-api-revoke-sub-d9b06bad-rsbbm\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:50.851972 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:50.851879 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:51.219348 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.219320 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm"] Nov 02 08:50:51.221634 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:50:51.221596 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode485365b_0cb4_440b_aa4d_64a58558cd15.slice/crio-bcd82f6a9634f4441d907e8be2e1792f0861f4b8310a87837843c3d98225cb20 WatchSource:0}: Error finding container bcd82f6a9634f4441d907e8be2e1792f0861f4b8310a87837843c3d98225cb20: Status 404 returned error can't find the container with id bcd82f6a9634f4441d907e8be2e1792f0861f4b8310a87837843c3d98225cb20 Nov 02 08:50:51.385427 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.385387 6160 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" podUID="78052f88-4743-4746-98ec-8f0ecb9d21cd" containerName="main" containerID="cri-o://770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850" gracePeriod=45 Nov 02 08:50:51.622927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.622901 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:50:51.704073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.704034 6160 generic.go:356] "Generic (PLEG): container finished" podID="78052f88-4743-4746-98ec-8f0ecb9d21cd" containerID="770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850" exitCode=0 Nov 02 08:50:51.704288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.704082 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" Nov 02 08:50:51.704288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.704128 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" event={"ID":"78052f88-4743-4746-98ec-8f0ecb9d21cd","Type":"ContainerDied","Data":"770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850"} Nov 02 08:50:51.704288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.704192 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv" event={"ID":"78052f88-4743-4746-98ec-8f0ecb9d21cd","Type":"ContainerDied","Data":"49b76842e63aa2cc859e22fa03d1cf0ef600d83adc1bc2d7eeccaf48aa7cece8"} Nov 02 08:50:51.704288 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.704217 6160 scope.go:122] "RemoveContainer" containerID="770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850" Nov 02 08:50:51.705726 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.705706 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" event={"ID":"e485365b-0cb4-440b-aa4d-64a58558cd15","Type":"ContainerStarted","Data":"a37df65fe897703750fd38387c8e68a64f2b81afed6ee5897a5970ca3b414c88"} Nov 02 08:50:51.705838 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.705747 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" event={"ID":"e485365b-0cb4-440b-aa4d-64a58558cd15","Type":"ContainerStarted","Data":"bcd82f6a9634f4441d907e8be2e1792f0861f4b8310a87837843c3d98225cb20"} Nov 02 08:50:51.713325 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.713304 6160 scope.go:122] "RemoveContainer" containerID="770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850" Nov 02 08:50:51.713611 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:50:51.713592 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850\": container with ID starting with 770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850 not found: ID does not exist" containerID="770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850" Nov 02 08:50:51.713677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.713617 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850"} err="failed to get container status \"770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850\": rpc error: code = NotFound desc = could not find container \"770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850\": container with ID starting with 770c8e24370bccfe7b57e25d9c0629e2b50fe3a69c41a1f696907491fa696850 not found: ID does not exist" Nov 02 08:50:51.817073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.816981 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78052f88-4743-4746-98ec-8f0ecb9d21cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/78052f88-4743-4746-98ec-8f0ecb9d21cd-tls-certs\") pod \"78052f88-4743-4746-98ec-8f0ecb9d21cd\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " Nov 02 08:50:51.817073 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817039 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-model-cache\") pod \"78052f88-4743-4746-98ec-8f0ecb9d21cd\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " Nov 02 08:50:51.817335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817084 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78052f88-4743-4746-98ec-8f0ecb9d21cd-kube-api-access-zlzlq\" (UniqueName: \"kubernetes.io/projected/78052f88-4743-4746-98ec-8f0ecb9d21cd-kube-api-access-zlzlq\") pod \"78052f88-4743-4746-98ec-8f0ecb9d21cd\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " Nov 02 08:50:51.817335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817117 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-dshm\") pod \"78052f88-4743-4746-98ec-8f0ecb9d21cd\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " Nov 02 08:50:51.817335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817188 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-home\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-home\") pod \"78052f88-4743-4746-98ec-8f0ecb9d21cd\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " Nov 02 08:50:51.817335 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817240 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-tmp-dir\") pod \"78052f88-4743-4746-98ec-8f0ecb9d21cd\" (UID: \"78052f88-4743-4746-98ec-8f0ecb9d21cd\") " Nov 02 08:50:51.817580 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817390 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-model-cache" pod "78052f88-4743-4746-98ec-8f0ecb9d21cd" (UID: "78052f88-4743-4746-98ec-8f0ecb9d21cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:51.817580 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817537 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-home" pod "78052f88-4743-4746-98ec-8f0ecb9d21cd" (UID: "78052f88-4743-4746-98ec-8f0ecb9d21cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:51.817688 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817617 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-tmp-dir" pod "78052f88-4743-4746-98ec-8f0ecb9d21cd" (UID: "78052f88-4743-4746-98ec-8f0ecb9d21cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:51.817907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817820 6160 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-model-cache\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.817907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817835 6160 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-home\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.817907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.817844 6160 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-tmp-dir\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.819499 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.819466 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78052f88-4743-4746-98ec-8f0ecb9d21cd-tls-certs" pod "78052f88-4743-4746-98ec-8f0ecb9d21cd" (UID: "78052f88-4743-4746-98ec-8f0ecb9d21cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:51.819876 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.819851 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78052f88-4743-4746-98ec-8f0ecb9d21cd-kube-api-access-zlzlq" pod "78052f88-4743-4746-98ec-8f0ecb9d21cd" (UID: "78052f88-4743-4746-98ec-8f0ecb9d21cd"). InnerVolumeSpecName "kube-api-access-zlzlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:51.819961 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.819913 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-dshm" pod "78052f88-4743-4746-98ec-8f0ecb9d21cd" (UID: "78052f88-4743-4746-98ec-8f0ecb9d21cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:50:51.886261 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.886201 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" podStartSLOduration=1.886170463 podStartE2EDuration="1.886170463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:50:51.88426251 +0000 UTC m=+2650.437766704" watchObservedRunningTime="2026-11-02 08:50:51.886170463 +0000 UTC m=+2650.439674660" Nov 02 08:50:51.918646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.918599 6160 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78052f88-4743-4746-98ec-8f0ecb9d21cd-tls-certs\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.918646 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.918642 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlzlq\" (UniqueName: \"kubernetes.io/projected/78052f88-4743-4746-98ec-8f0ecb9d21cd-kube-api-access-zlzlq\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:51.918897 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:51.918658 6160 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78052f88-4743-4746-98ec-8f0ecb9d21cd-dshm\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:52.118222 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:52.118191 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv"] Nov 02 08:50:52.119978 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:52.119955 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d0628e/distinct-2-w1-d0628e-kserve-56d664698d-klwzv"] Nov 02 08:50:52.713459 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:52.713407 6160 generic.go:356] "Generic (PLEG): container finished" podID="e485365b-0cb4-440b-aa4d-64a58558cd15" containerID="a37df65fe897703750fd38387c8e68a64f2b81afed6ee5897a5970ca3b414c88" exitCode=0 Nov 02 08:50:52.713973 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:52.713484 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" event={"ID":"e485365b-0cb4-440b-aa4d-64a58558cd15","Type":"ContainerDied","Data":"a37df65fe897703750fd38387c8e68a64f2b81afed6ee5897a5970ca3b414c88"} Nov 02 08:50:53.583819 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:53.583784 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78052f88-4743-4746-98ec-8f0ecb9d21cd" path="/var/lib/kubelet/pods/78052f88-4743-4746-98ec-8f0ecb9d21cd/volumes" Nov 02 08:50:53.850562 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:53.850536 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:53.935000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:53.934964 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-xks-ca\") pod \"e485365b-0cb4-440b-aa4d-64a58558cd15\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " Nov 02 08:50:53.935211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:53.935012 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-openshift-ca\") pod \"e485365b-0cb4-440b-aa4d-64a58558cd15\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " Nov 02 08:50:53.935211 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:53.935044 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e485365b-0cb4-440b-aa4d-64a58558cd15-kube-api-access-95vqw\" (UniqueName: \"kubernetes.io/projected/e485365b-0cb4-440b-aa4d-64a58558cd15-kube-api-access-95vqw\") pod \"e485365b-0cb4-440b-aa4d-64a58558cd15\" (UID: \"e485365b-0cb4-440b-aa4d-64a58558cd15\") " Nov 02 08:50:53.935421 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:53.935396 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-openshift-ca" pod "e485365b-0cb4-440b-aa4d-64a58558cd15" (UID: "e485365b-0cb4-440b-aa4d-64a58558cd15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:53.937368 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:53.937345 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e485365b-0cb4-440b-aa4d-64a58558cd15-kube-api-access-95vqw" pod "e485365b-0cb4-440b-aa4d-64a58558cd15" (UID: "e485365b-0cb4-440b-aa4d-64a58558cd15"). InnerVolumeSpecName "kube-api-access-95vqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:53.937437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:53.937377 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-xks-ca" pod "e485365b-0cb4-440b-aa4d-64a58558cd15" (UID: "e485365b-0cb4-440b-aa4d-64a58558cd15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:54.036743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:54.036698 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-xks-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:54.036743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:54.036735 6160 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e485365b-0cb4-440b-aa4d-64a58558cd15-maas-api-openshift-ca\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:54.036743 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:54.036746 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95vqw\" (UniqueName: \"kubernetes.io/projected/e485365b-0cb4-440b-aa4d-64a58558cd15-kube-api-access-95vqw\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:50:54.726209 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:54.726153 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" event={"ID":"e485365b-0cb4-440b-aa4d-64a58558cd15","Type":"ContainerDied","Data":"bcd82f6a9634f4441d907e8be2e1792f0861f4b8310a87837843c3d98225cb20"} Nov 02 08:50:54.726425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:54.726213 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b06bad-rsbbm" Nov 02 08:50:54.726425 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:54.726225 6160 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcd82f6a9634f4441d907e8be2e1792f0861f4b8310a87837843c3d98225cb20" Nov 02 08:50:57.144943 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:50:57.144897 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" podUID="84488f2a-6ede-40e9-a394-2db0a6f25669" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:51:06.061952 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:06.061911 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:07.145525 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:07.145480 6160 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" podUID="84488f2a-6ede-40e9-a394-2db0a6f25669" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:51:07.586616 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:07.586579 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:07.895674 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:07.895581 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:08.138257 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.138217 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d"] Nov 02 08:51:08.183700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.183668 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d0628e-sqx2d"] Nov 02 08:51:08.543858 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.543827 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:08.749501 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.749466 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:51:08.800404 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.800328 6160 generic.go:356] "Generic (PLEG): container finished" podID="84488f2a-6ede-40e9-a394-2db0a6f25669" containerID="da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75" exitCode=0 Nov 02 08:51:08.800553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.800417 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" event={"ID":"84488f2a-6ede-40e9-a394-2db0a6f25669","Type":"ContainerDied","Data":"da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75"} Nov 02 08:51:08.800553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.800436 6160 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" Nov 02 08:51:08.800553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.800460 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw" event={"ID":"84488f2a-6ede-40e9-a394-2db0a6f25669","Type":"ContainerDied","Data":"d24e3c9f1ac98aac485f117f5f574d147bdbb47f0e71ecf07e6151e4a067f2be"} Nov 02 08:51:08.800553 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.800486 6160 scope.go:122] "RemoveContainer" containerID="da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75" Nov 02 08:51:08.810037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.810015 6160 scope.go:122] "RemoveContainer" containerID="da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75" Nov 02 08:51:08.810340 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:51:08.810319 6160 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75\": container with ID starting with da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75 not found: ID does not exist" containerID="da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75" Nov 02 08:51:08.810402 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.810353 6160 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75"} err="failed to get container status \"da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75\": rpc error: code = NotFound desc = could not find container \"da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75\": container with ID starting with da55fcc13b39f2fbe5d1c26d00105de31294ab688b9fd7f261379b83b4055b75 not found: ID does not exist" Nov 02 08:51:08.876460 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.876431 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84488f2a-6ede-40e9-a394-2db0a6f25669-kube-api-access-r4f7h\" (UniqueName: \"kubernetes.io/projected/84488f2a-6ede-40e9-a394-2db0a6f25669-kube-api-access-r4f7h\") pod \"84488f2a-6ede-40e9-a394-2db0a6f25669\" (UID: \"84488f2a-6ede-40e9-a394-2db0a6f25669\") " Nov 02 08:51:08.876645 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.876564 6160 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84488f2a-6ede-40e9-a394-2db0a6f25669-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84488f2a-6ede-40e9-a394-2db0a6f25669-plugins-config-volume\") pod \"84488f2a-6ede-40e9-a394-2db0a6f25669\" (UID: \"84488f2a-6ede-40e9-a394-2db0a6f25669\") " Nov 02 08:51:08.876929 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.876907 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84488f2a-6ede-40e9-a394-2db0a6f25669-plugins-config-volume" pod "84488f2a-6ede-40e9-a394-2db0a6f25669" (UID: "84488f2a-6ede-40e9-a394-2db0a6f25669"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:51:08.878977 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.878954 6160 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84488f2a-6ede-40e9-a394-2db0a6f25669-kube-api-access-r4f7h" pod "84488f2a-6ede-40e9-a394-2db0a6f25669" (UID: "84488f2a-6ede-40e9-a394-2db0a6f25669"). InnerVolumeSpecName "kube-api-access-r4f7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:51:08.977851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.977813 6160 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84488f2a-6ede-40e9-a394-2db0a6f25669-plugins-config-volume\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:08.977851 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:08.977845 6160 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f7h\" (UniqueName: \"kubernetes.io/projected/84488f2a-6ede-40e9-a394-2db0a6f25669-kube-api-access-r4f7h\") on node \"ip-10-0-136-103.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 08:51:09.211916 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:09.211882 6160 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw"] Nov 02 08:51:09.215576 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:09.215547 6160 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d0628e-6f566bc799-h56sw"] Nov 02 08:51:09.582102 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:09.582063 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396ea774-88ee-46b2-9b3b-ad6bc9b686ee" path="/var/lib/kubelet/pods/396ea774-88ee-46b2-9b3b-ad6bc9b686ee/volumes" Nov 02 08:51:09.582520 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:09.582506 6160 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84488f2a-6ede-40e9-a394-2db0a6f25669" path="/var/lib/kubelet/pods/84488f2a-6ede-40e9-a394-2db0a6f25669/volumes" Nov 02 08:51:10.520898 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:51:10.520860 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:51:13.416788 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:13.416748 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:13.618132 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:13.618093 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:14.475548 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:14.475511 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:15.329967 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:15.329937 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:15.508268 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:15.508223 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:15.685198 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:15.685150 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:15.844598 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:15.844563 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:17.920003 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:17.919976 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lwl75_d2e8ac17-bdcf-4895-acd8-105b444731fc/kuadrant-console-plugin/0.log" Nov 02 08:51:19.090000 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:19.089967 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:19.264783 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:19.264744 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:20.226680 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:20.226644 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:20.265035 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:20.264996 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c9959c9bb-l695c_74e96d56-8e33-4351-814b-3d9781be59fd/manager/0.log" Nov 02 08:51:22.076528 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:22.076492 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fbqk9_1fd6f36b-0fce-4b8a-ae80-f163a73f87c9/server/0.log" Nov 02 08:51:32.319907 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:32.319860 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:40.521257 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:51:40.521216 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:51:42.043387 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:42.043358 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:51:42.048668 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:51:42.048647 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gvxtz_a99d10da-d327-4799-b1c2-ef2c23cb5aa1/extract/0.log" Nov 02 08:52:03.767833 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:03.767798 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c9959c9bb-l695c_74e96d56-8e33-4351-814b-3d9781be59fd/manager/0.log" Nov 02 08:52:05.199612 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:05.199582 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fbqk9_1fd6f36b-0fce-4b8a-ae80-f163a73f87c9/server/0.log" Nov 02 08:52:07.535841 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:07.535810 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d9b06bad-rsbbm_e485365b-0cb4-440b-aa4d-64a58558cd15/revoke-keys/0.log" Nov 02 08:52:10.088556 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:10.088531 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lwl75_d2e8ac17-bdcf-4895-acd8-105b444731fc/kuadrant-console-plugin/0.log" Nov 02 08:52:10.249832 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:10.249797 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:10.440856 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:10.440818 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:10.520868 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:52:10.520833 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:52:12.758087 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:12.758050 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ddcd68ff8-ckxc2_5bfaa88b-36fe-4db7-8638-a30f7a0b7b87/kube-auth-proxy/0.log" Nov 02 08:52:14.327683 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:14.327651 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-j82zb_2087bcd1-2ad8-4ce0-a356-99720eb3c494/payload-pre-processing/0.log" Nov 02 08:52:14.549590 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:14.549554 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-4h2lv_c7297abc-ee50-4bd9-8065-99c7cd95357f/payload-processing/0.log" Nov 02 08:52:15.806857 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:15.806826 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zwwbc_c213051c-75c6-4e21-95cb-3870a450b435/main/0.log" Nov 02 08:52:16.571723 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:16.571695 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-rkx8l_ec12dd43-98b4-47b7-adb0-c6aa56108563/main/0.log" Nov 02 08:52:16.655234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:16.655194 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:16.839564 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:16.839487 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pb27h_3d35ef8c-274f-4e8d-97f4-c0960dd718b6/main/0.log" Nov 02 08:52:17.373108 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:17.373056 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-j9zww_50909652-5fa6-4f37-9d22-6122af29d8b1/main/0.log" Nov 02 08:52:26.818104 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:26.818068 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:26.974734 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:26.974699 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:28.095331 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095295 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgtvq/must-gather-hl545"] Nov 02 08:52:28.095731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095632 6160 cpu_manager.go:414] "containerMap: removing container" podUID="78052f88-4743-4746-98ec-8f0ecb9d21cd" containerName="main" Nov 02 08:52:28.095731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095643 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78052f88-4743-4746-98ec-8f0ecb9d21cd" containerName="main" Nov 02 08:52:28.095731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095665 6160 cpu_manager.go:414] "containerMap: removing container" podUID="e485365b-0cb4-440b-aa4d-64a58558cd15" containerName="revoke-keys" Nov 02 08:52:28.095731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095671 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e485365b-0cb4-440b-aa4d-64a58558cd15" containerName="revoke-keys" Nov 02 08:52:28.095731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095691 6160 cpu_manager.go:414] "containerMap: removing container" podUID="84488f2a-6ede-40e9-a394-2db0a6f25669" containerName="payload-pre-processing" Nov 02 08:52:28.095731 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095697 6160 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84488f2a-6ede-40e9-a394-2db0a6f25669" containerName="payload-pre-processing" Nov 02 08:52:28.095927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095747 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="84488f2a-6ede-40e9-a394-2db0a6f25669" containerName="payload-pre-processing" Nov 02 08:52:28.095927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095756 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="78052f88-4743-4746-98ec-8f0ecb9d21cd" containerName="main" Nov 02 08:52:28.095927 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.095767 6160 memory_manager.go:365] "RemoveStaleState removing state" podUID="e485365b-0cb4-440b-aa4d-64a58558cd15" containerName="revoke-keys" Nov 02 08:52:28.099094 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.099075 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgtvq/must-gather-hl545" Nov 02 08:52:28.126986 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.126959 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:28.147488 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.147459 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgtvq\"/\"openshift-service-ca.crt\"" Nov 02 08:52:28.147669 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.147520 6160 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hgtvq\"/\"default-dockercfg-p95c4\"" Nov 02 08:52:28.193925 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.193890 6160 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgtvq\"/\"kube-root-ca.crt\"" Nov 02 08:52:28.195856 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.195835 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgtvq/must-gather-hl545"] Nov 02 08:52:28.201782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.201755 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69dff7f6-2434-4e72-9154-25320724ce9c-must-gather-output\") pod \"must-gather-hl545\" (UID: \"69dff7f6-2434-4e72-9154-25320724ce9c\") " pod="openshift-must-gather-hgtvq/must-gather-hl545" Nov 02 08:52:28.201975 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.201952 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fxsj\" (UniqueName: \"kubernetes.io/projected/69dff7f6-2434-4e72-9154-25320724ce9c-kube-api-access-2fxsj\") pod \"must-gather-hl545\" (UID: \"69dff7f6-2434-4e72-9154-25320724ce9c\") " pod="openshift-must-gather-hgtvq/must-gather-hl545" Nov 02 08:52:28.294199 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.294135 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:28.303241 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.303213 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fxsj\" (UniqueName: \"kubernetes.io/projected/69dff7f6-2434-4e72-9154-25320724ce9c-kube-api-access-2fxsj\") pod \"must-gather-hl545\" (UID: \"69dff7f6-2434-4e72-9154-25320724ce9c\") " pod="openshift-must-gather-hgtvq/must-gather-hl545" Nov 02 08:52:28.303380 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.303257 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69dff7f6-2434-4e72-9154-25320724ce9c-must-gather-output\") pod \"must-gather-hl545\" (UID: \"69dff7f6-2434-4e72-9154-25320724ce9c\") " pod="openshift-must-gather-hgtvq/must-gather-hl545" Nov 02 08:52:28.303631 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.303616 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69dff7f6-2434-4e72-9154-25320724ce9c-must-gather-output\") pod \"must-gather-hl545\" (UID: \"69dff7f6-2434-4e72-9154-25320724ce9c\") " pod="openshift-must-gather-hgtvq/must-gather-hl545" Nov 02 08:52:28.358660 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.358587 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fxsj\" (UniqueName: \"kubernetes.io/projected/69dff7f6-2434-4e72-9154-25320724ce9c-kube-api-access-2fxsj\") pod \"must-gather-hl545\" (UID: \"69dff7f6-2434-4e72-9154-25320724ce9c\") " pod="openshift-must-gather-hgtvq/must-gather-hl545" Nov 02 08:52:28.412729 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.412688 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgtvq/must-gather-hl545" Nov 02 08:52:28.778417 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:28.778386 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgtvq/must-gather-hl545"] Nov 02 08:52:28.781143 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:52:28.781098 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69dff7f6_2434_4e72_9154_25320724ce9c.slice/crio-524a3c4bdbf352a766a751549322d2f2b0b929f3dbb8b38533e1d515b105dc0a WatchSource:0}: Error finding container 524a3c4bdbf352a766a751549322d2f2b0b929f3dbb8b38533e1d515b105dc0a: Status 404 returned error can't find the container with id 524a3c4bdbf352a766a751549322d2f2b0b929f3dbb8b38533e1d515b105dc0a Nov 02 08:52:29.211298 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:29.211261 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgtvq/must-gather-hl545" event={"ID":"69dff7f6-2434-4e72-9154-25320724ce9c","Type":"ContainerStarted","Data":"524a3c4bdbf352a766a751549322d2f2b0b929f3dbb8b38533e1d515b105dc0a"} Nov 02 08:52:29.921689 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:29.921649 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:31.225080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:31.225030 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgtvq/must-gather-hl545" event={"ID":"69dff7f6-2434-4e72-9154-25320724ce9c","Type":"ContainerStarted","Data":"a098cddc094af938dc82218d28e78fd5efe9d1980c75dddee5e042ab257e8d9f"} Nov 02 08:52:31.225080 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:31.225080 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgtvq/must-gather-hl545" event={"ID":"69dff7f6-2434-4e72-9154-25320724ce9c","Type":"ContainerStarted","Data":"7098f1df83adcd6c5ccfc42f5e608bf68214e19007a0f6bda6ab325732b921a9"} Nov 02 08:52:31.332353 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:31.332287 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hgtvq/must-gather-hl545" podStartSLOduration=1.960135444 podStartE2EDuration="3.332273294s" totalImagesPullingTime="1.37213785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:52:31.329929168 +0000 UTC m=+2749.883433366" watchObservedRunningTime="2026-11-02 08:52:31.332273294 +0000 UTC m=+2749.885777501" Nov 02 08:52:31.881291 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:31.881245 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:32.647768 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:32.647733 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:37.819037 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:37.818997 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xs697_f4015660-eec1-47d8-8432-de3b74803f6a/global-pull-secret-syncer/0.log" Nov 02 08:52:39.755575 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:39.755545 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" Nov 02 08:52:40.544084 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:52:40.538577 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor" Nov 02 08:52:41.494683 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:41.494087 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-103.us-west-2.compute.internal_fdbcb27b80ac4a4b19ed5f7d7dc1c25f/haproxy/0.log" Nov 02 08:52:42.428782 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:42.428742 6160 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfqx_8e01da69-1f87-48ec-9edf-b73ccbf0b244/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:47.176602 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.176552 6160 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56"] Nov 02 08:52:47.183992 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.183963 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.264120 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.264084 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56"] Nov 02 08:52:47.282826 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.282792 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-proc\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.283013 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.282840 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzkz\" (UniqueName: \"kubernetes.io/projected/5c2550d1-2129-425d-82d6-cf0ff83e11f7-kube-api-access-knzkz\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.283013 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.282994 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-podres\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.283132 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.283038 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-sys\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.283237 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.283216 6160 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-lib-modules\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.384204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.384140 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-podres\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.384204 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.384207 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-sys\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.384437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.384249 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-podres\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.384437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.384263 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-lib-modules\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.384437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.384320 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-lib-modules\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.384437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.384324 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-proc\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.384437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.384336 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-sys\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.384437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.384358 6160 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzkz\" (UniqueName: \"kubernetes.io/projected/5c2550d1-2129-425d-82d6-cf0ff83e11f7-kube-api-access-knzkz\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.384437 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.384367 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c2550d1-2129-425d-82d6-cf0ff83e11f7-proc\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.438946 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.438873 6160 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzkz\" (UniqueName: \"kubernetes.io/projected/5c2550d1-2129-425d-82d6-cf0ff83e11f7-kube-api-access-knzkz\") pod \"perf-node-gather-daemonset-9ls56\" (UID: \"5c2550d1-2129-425d-82d6-cf0ff83e11f7\") " pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.500677 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.500644 6160 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:47.869221 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.869118 6160 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56"] Nov 02 08:52:47.873586 ip-10-0-136-103 kubenswrapper[6160]: W1102 08:52:47.873551 6160 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c2550d1_2129_425d_82d6_cf0ff83e11f7.slice/crio-0e6a0398c97c8926a2dc42d7338ec1e862741c4b02687fc46ce054f5348311af WatchSource:0}: Error finding container 0e6a0398c97c8926a2dc42d7338ec1e862741c4b02687fc46ce054f5348311af: Status 404 returned error can't find the container with id 0e6a0398c97c8926a2dc42d7338ec1e862741c4b02687fc46ce054f5348311af Nov 02 08:52:47.935240 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:47.935216 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lwl75_d2e8ac17-bdcf-4895-acd8-105b444731fc/kuadrant-console-plugin/0.log" Nov 02 08:52:48.358311 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:48.358276 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" event={"ID":"5c2550d1-2129-425d-82d6-cf0ff83e11f7","Type":"ContainerStarted","Data":"93e4616a321851bde8c885c70e243b9f2beb6a171415d0e904bd592d58f39e09"} Nov 02 08:52:48.358763 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:48.358328 6160 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" event={"ID":"5c2550d1-2129-425d-82d6-cf0ff83e11f7","Type":"ContainerStarted","Data":"0e6a0398c97c8926a2dc42d7338ec1e862741c4b02687fc46ce054f5348311af"} Nov 02 08:52:48.358763 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:48.358386 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:48.466472 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:48.466398 6160 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" podStartSLOduration=1.46637858 podStartE2EDuration="1.46637858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 08:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:52:48.465768009 +0000 UTC m=+2767.019272206" watchObservedRunningTime="2026-11-02 08:52:48.46637858 +0000 UTC m=+2767.019882778" Nov 02 08:52:54.375032 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:54.375001 6160 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hgtvq/perf-node-gather-daemonset-9ls56" Nov 02 08:52:57.580863 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:57.580831 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m89dq_b5f83ac7-e0a5-47db-bf75-f72a32394313/node-exporter/0.log" Nov 02 08:52:57.987072 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:57.987039 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m89dq_b5f83ac7-e0a5-47db-bf75-f72a32394313/kube-rbac-proxy/0.log" Nov 02 08:52:58.043873 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:58.043727 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c9959c9bb-l695c_74e96d56-8e33-4351-814b-3d9781be59fd/manager/0.log" Nov 02 08:52:58.394234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:58.394148 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m89dq_b5f83ac7-e0a5-47db-bf75-f72a32394313/init-textfile/0.log" Nov 02 08:52:59.179295 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:52:59.179257 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fbqk9_1fd6f36b-0fce-4b8a-ae80-f163a73f87c9/server/0.log" Nov 02 08:53:02.714566 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:02.714536 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13f5952e-7c2f-41b1-919e-725db0078f60/prometheus/0.log" Nov 02 08:53:03.123642 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:03.123603 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13f5952e-7c2f-41b1-919e-725db0078f60/config-reloader/0.log" Nov 02 08:53:03.531363 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:03.531335 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13f5952e-7c2f-41b1-919e-725db0078f60/thanos-sidecar/0.log" Nov 02 08:53:03.848234 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:03.848105 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c9959c9bb-l695c_74e96d56-8e33-4351-814b-3d9781be59fd/manager/0.log" Nov 02 08:53:03.939044 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:03.939001 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13f5952e-7c2f-41b1-919e-725db0078f60/kube-rbac-proxy-web/0.log" Nov 02 08:53:04.347700 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:04.347651 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13f5952e-7c2f-41b1-919e-725db0078f60/kube-rbac-proxy/0.log" Nov 02 08:53:04.757853 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:04.757820 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13f5952e-7c2f-41b1-919e-725db0078f60/kube-rbac-proxy-thanos/0.log" Nov 02 08:53:05.067808 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:05.066219 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fbqk9_1fd6f36b-0fce-4b8a-ae80-f163a73f87c9/server/0.log" Nov 02 08:53:05.169595 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:05.169551 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13f5952e-7c2f-41b1-919e-725db0078f60/init-config-reloader/0.log" Nov 02 08:53:05.588745 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:05.588716 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bpdm6_68679646-df21-499f-b5dd-c5b6ceb12314/prometheus-operator/0.log" Nov 02 08:53:05.991966 ip-10-0-136-103 kubenswrapper[6160]: I1102 08:53:05.991929 6160 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bpdm6_68679646-df21-499f-b5dd-c5b6ceb12314/kube-rbac-proxy/0.log" Nov 02 08:53:10.539303 ip-10-0-136-103 kubenswrapper[6160]: E1102 08:53:10.539232 6160 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kck72" containerName="extractor"